How to check the total database size?
The TrendMiner disk size needs to store a lot of data. The most significant data stored on the TrendMiner server is:
The OS (Linux).
The TrendMiner software, with mainly:
Index data synced from external data sources (stored in files).
User data (stored in a database).
This article explains how to measure the size of the user data which is stored in the database.
To get the total disk size of the database data:
Create a manual database backup in Edge Manager -> Backup and Restore -> Database only.
When finished, download the backup to your local machine. The expected file name format:
trendminer-<datetime>.tar.gz.Decompress the .gz file using third-party software or with Powershell by running the following command:
& { $src = (Read-Host 'Path to .gz file').Trim('"', "'", ' ') $src = (Resolve-Path -LiteralPath $src).Path; $dst = $src -replace '\.gz$', '' $in = [IO.File]::OpenRead($src); $out = [IO.File]::Create($dst) $gz = New-Object IO.Compression.GZipStream($in, [IO.Compression.CompressionMode]::Decompress) $buf = New-Object byte[] (1MB); $total = [long]0; $sw = [Diagnostics.Stopwatch]::StartNew(); $last = 0 try { while (($n = $gz.Read($buf, 0, $buf.Length)) -gt 0) { $out.Write($buf, 0, $n); $total += $n if ($sw.ElapsedMilliseconds - $last -ge 500) { $last = $sw.ElapsedMilliseconds $pct = [math]::Min(100, [int](100 * $in.Position / $in.Length)) Write-Progress -Activity "Decompressing $(Split-Path $src -Leaf)" -PercentComplete $pct ` -Status ("{0}% - {1:N2} GB written" -f $pct, ($total / 1GB)) } } } finally { $out.Dispose(); $gz.Dispose(); $in.Dispose(); Write-Progress -Activity "Decompressing" -Completed } "Wrote $dst ({0:N2} GB, {1:N0} seconds)" -f ($total / 1GB), $sw.Elapsed.TotalSeconds }When requested, enter the full path name for the
trendminer-<datetime>.tar.gzfile.When decompression is done a
trendminer-<datetime>.tarfile will be created.Untar the .tar file using third-party software or with Powershell by running the following command:
& { $src = (Read-Host 'Path to .tar file').Trim('"', "'", ' ') $src = (Resolve-Path -LiteralPath $src).Path; $dest = $src -replace '\.tar$', ''; if ($dest -eq $src) { $dest += '_extracted' } $in = [IO.File]::OpenRead($src); $h = New-Object byte[] 512; $buf = New-Object byte[] (1MB) $p = @{ last = 0; sw = [Diagnostics.Stopwatch]::StartNew() }; $count = 0; $long = $null $str = { param($b, $o, $l) $s = [Text.Encoding]::UTF8.GetString($b, $o, $l); $i = $s.IndexOf([char]0); if ($i -ge 0) { $s.Substring(0, $i) } else { $s } } $read = { param([long]$cnt, $o) while ($cnt -gt 0) { $n = $in.Read($buf, 0, [int][math]::Min([long]$buf.Length, $cnt)); if ($n -le 0) { throw 'Unexpected end of archive' } if ($o) { $o.Write($buf, 0, $n) }; $cnt -= $n if ($p.sw.ElapsedMilliseconds - $p.last -ge 500) { $p.last = $p.sw.ElapsedMilliseconds $pct = [int](100 * $in.Position / $in.Length); Write-Progress -Activity "Extracting $(Split-Path $src -Leaf)" -Status "$pct%" -PercentComplete $pct } } } try { while ($in.Read($h, 0, 512) -eq 512 -and $h[0] -ne 0) { $t = [char]$h[156]; $sz = (& $str $h 124 12).Trim(); $size = if ($sz) { [Convert]::ToInt64($sz, 8) } else { [long]0 } $name = & $str $h 0 100; $pre = & $str $h 345 155 if ($pre -and (& $str $h 257 5) -eq 'ustar') { $name = "$pre/$name" } if ($long) { $name = $long; $long = $null } if ($t -ceq 'L' -or $t -ceq 'x') { $ms = New-Object IO.MemoryStream; & $read $size $ms; $txt = & $str $ms.ToArray() 0 ([int]$size) if ($t -ceq 'L') { $long = $txt } elseif ($txt -match '(?m)^\d+ path=(.*)$') { $long = $Matches[1] } } else { $rel = (($name -split '[\\/]') | Where-Object { $_ -and $_ -ne '.' -and $_ -ne '..' } | ForEach-Object { $_ -replace '[<>:"|?*]', '_' }) -join '\' $path = if ($rel) { Join-Path $dest $rel } if ($t -ceq '5') { if ($path) { [void][IO.Directory]::CreateDirectory($path) } } elseif ($path -and ($t -ceq '0' -or $t -ceq '7' -or $h[156] -eq 0)) { [void][IO.Directory]::CreateDirectory([IO.Path]::GetDirectoryName($path)) $out = [IO.File]::Create($path); try { & $read $size $out } finally { $out.Dispose() }; $count++ } else { & $read $size $null } } $pad = (512 - $size % 512) % 512; if ($pad) { & $read $pad $null } } } finally { $in.Dispose(); Write-Progress -Activity 'Extracting' -Completed } "Extracted $count files to $dest ({0:N0} seconds)" -f $p.sw.Elapsed.TotalSeconds }When requested, enter the full path name for the
trendminer-<datetime>.tarfile.When decompression is done a
trendminer-<datetime>folder will be created. which contains the compressed TrendMiner databases.Run the following Powershell command to get the total uncompressed disk size from all databases:
& { $folder = (Read-Host 'Folder with .gz files').Trim('"', "'", ' ') $files = @(Get-ChildItem -LiteralPath $folder -File | Where-Object Extension -in '.gz','.tgz') $buf = New-Object byte[] (1MB); $total = [long]0; $sw = [Diagnostics.Stopwatch]::StartNew(); $last = 0 for ($i = 0; $i -lt $files.Count; $i++) { $in = [IO.File]::OpenRead($files[$i].FullName) $gz = New-Object IO.Compression.GZipStream($in, [IO.Compression.CompressionMode]::Decompress) try { while (($n = $gz.Read($buf, 0, $buf.Length)) -gt 0) { $total += $n if ($sw.ElapsedMilliseconds - $last -ge 500) { $last = $sw.ElapsedMilliseconds Write-Progress -Activity "Measuring uncompressed size" -PercentComplete ([int](100 * ($i + $in.Position / $in.Length) / $files.Count)) ` -Status ("File {0}/{1}: {2} - {3:N2} GB so far" -f ($i+1), $files.Count, $files[$i].Name, ($total/1GB)) } } } finally { $gz.Dispose(); $in.Dispose() } } Write-Progress -Activity "Measuring uncompressed size" -Completed "{0} files, {1:N2} GB uncompressed ({2:N0} seconds)" -f $files.Count, ($total/1GB), $sw.Elapsed.TotalSeconds }When requested, enter the full path name for the
trendminer-<datetime>folder.The result shows the total uncompressed disk space used by TrendMiner databases.
