Skip to main content

Documentation Portal

How to check the total database size?

The TrendMiner disk size needs to store a lot of data. The most significant data stored on the TrendMiner server is:

  • The OS (Linux).

  • The TrendMiner software, with mainly:

    • Index data synced from external data sources (stored in files).

    • User data (stored in a database).

This article explains how to measure the size of the user data which is stored in the database.

To get the total disk size of the database data:

  1. Create a manual database backup in Edge Manager -> Backup and Restore -> Database only.

  2. When finished, download the backup to your local machine. The expected file name format: trendminer-<datetime>.tar.gz.

  3. Decompress the .gz file using third-party software or with Powershell by running the following command:

    & { $src = (Read-Host 'Path to .gz file').Trim('"', "'", ' ')
      $src = (Resolve-Path -LiteralPath $src).Path; $dst = $src -replace '\.gz$', ''
      $in = [IO.File]::OpenRead($src); $out = [IO.File]::Create($dst)
      $gz = New-Object IO.Compression.GZipStream($in, [IO.Compression.CompressionMode]::Decompress)
      $buf = New-Object byte[] (1MB); $total = [long]0; $sw = [Diagnostics.Stopwatch]::StartNew(); $last = 0
      try { while (($n = $gz.Read($buf, 0, $buf.Length)) -gt 0) { $out.Write($buf, 0, $n); $total += $n
          if ($sw.ElapsedMilliseconds - $last -ge 500) { $last = $sw.ElapsedMilliseconds
            $pct = [math]::Min(100, [int](100 * $in.Position / $in.Length))
            Write-Progress -Activity "Decompressing $(Split-Path $src -Leaf)" -PercentComplete $pct `
              -Status ("{0}% - {1:N2} GB written" -f $pct, ($total / 1GB)) } } }
      finally { $out.Dispose(); $gz.Dispose(); $in.Dispose(); Write-Progress -Activity "Decompressing" -Completed }
      "Wrote $dst ({0:N2} GB, {1:N0} seconds)" -f ($total / 1GB), $sw.Elapsed.TotalSeconds }

    When requested, enter the full path name for the trendminer-<datetime>.tar.gz file.

    When decompression is done a trendminer-<datetime>.tar file will be created.

  4. Untar the .tar file using third-party software or with Powershell by running the following command:

    & { $src = (Read-Host 'Path to .tar file').Trim('"', "'", ' ')
      $src = (Resolve-Path -LiteralPath $src).Path; $dest = $src -replace '\.tar$', ''; if ($dest -eq $src) { $dest += '_extracted' }
      $in = [IO.File]::OpenRead($src); $h = New-Object byte[] 512; $buf = New-Object byte[] (1MB)
      $p = @{ last = 0; sw = [Diagnostics.Stopwatch]::StartNew() }; $count = 0; $long = $null
      $str = { param($b, $o, $l) $s = [Text.Encoding]::UTF8.GetString($b, $o, $l); $i = $s.IndexOf([char]0); if ($i -ge 0) { $s.Substring(0, $i) } else { $s } }
      $read = { param([long]$cnt, $o) while ($cnt -gt 0) {
          $n = $in.Read($buf, 0, [int][math]::Min([long]$buf.Length, $cnt)); if ($n -le 0) { throw 'Unexpected end of archive' }
          if ($o) { $o.Write($buf, 0, $n) }; $cnt -= $n
          if ($p.sw.ElapsedMilliseconds - $p.last -ge 500) { $p.last = $p.sw.ElapsedMilliseconds
            $pct = [int](100 * $in.Position / $in.Length); Write-Progress -Activity "Extracting $(Split-Path $src -Leaf)" -Status "$pct%" -PercentComplete $pct } } }
      try { while ($in.Read($h, 0, 512) -eq 512 -and $h[0] -ne 0) {
          $t = [char]$h[156]; $sz = (& $str $h 124 12).Trim(); $size = if ($sz) { [Convert]::ToInt64($sz, 8) } else { [long]0 }
          $name = & $str $h 0 100; $pre = & $str $h 345 155
          if ($pre -and (& $str $h 257 5) -eq 'ustar') { $name = "$pre/$name" }
          if ($long) { $name = $long; $long = $null }
          if ($t -ceq 'L' -or $t -ceq 'x') {
            $ms = New-Object IO.MemoryStream; & $read $size $ms; $txt = & $str $ms.ToArray() 0 ([int]$size)
            if ($t -ceq 'L') { $long = $txt } elseif ($txt -match '(?m)^\d+ path=(.*)$') { $long = $Matches[1] } }
          else {
            $rel = (($name -split '[\\/]') | Where-Object { $_ -and $_ -ne '.' -and $_ -ne '..' } | ForEach-Object { $_ -replace '[<>:"|?*]', '_' }) -join '\'
            $path = if ($rel) { Join-Path $dest $rel }
            if ($t -ceq '5') { if ($path) { [void][IO.Directory]::CreateDirectory($path) } }
            elseif ($path -and ($t -ceq '0' -or $t -ceq '7' -or $h[156] -eq 0)) {
              [void][IO.Directory]::CreateDirectory([IO.Path]::GetDirectoryName($path))
              $out = [IO.File]::Create($path); try { & $read $size $out } finally { $out.Dispose() }; $count++ }
            else { & $read $size $null } }
          $pad = (512 - $size % 512) % 512; if ($pad) { & $read $pad $null } } }
      finally { $in.Dispose(); Write-Progress -Activity 'Extracting' -Completed }
      "Extracted $count files to $dest ({0:N0} seconds)" -f $p.sw.Elapsed.TotalSeconds }

    When requested, enter the full path name for the trendminer-<datetime>.tar file.

    When decompression is done a trendminer-<datetime> folder will be created. which contains the compressed TrendMiner databases.

  5. Run the following Powershell command to get the total uncompressed disk size from all databases:

    & { $folder = (Read-Host 'Folder with .gz files').Trim('"', "'", ' ')
      $files = @(Get-ChildItem -LiteralPath $folder -File | Where-Object Extension -in '.gz','.tgz')
      $buf = New-Object byte[] (1MB); $total = [long]0; $sw = [Diagnostics.Stopwatch]::StartNew(); $last = 0
      for ($i = 0; $i -lt $files.Count; $i++) {
        $in = [IO.File]::OpenRead($files[$i].FullName)
        $gz = New-Object IO.Compression.GZipStream($in, [IO.Compression.CompressionMode]::Decompress)
        try { while (($n = $gz.Read($buf, 0, $buf.Length)) -gt 0) { $total += $n
          if ($sw.ElapsedMilliseconds - $last -ge 500) { $last = $sw.ElapsedMilliseconds
            Write-Progress -Activity "Measuring uncompressed size" -PercentComplete ([int](100 * ($i + $in.Position / $in.Length) / $files.Count)) `
              -Status ("File {0}/{1}: {2} - {3:N2} GB so far" -f ($i+1), $files.Count, $files[$i].Name, ($total/1GB)) } } }
        finally { $gz.Dispose(); $in.Dispose() } }
      Write-Progress -Activity "Measuring uncompressed size" -Completed
      "{0} files, {1:N2} GB uncompressed ({2:N0} seconds)" -f $files.Count, ($total/1GB), $sw.Elapsed.TotalSeconds }

    When requested, enter the full path name for the trendminer-<datetime> folder.

    The result shows the total uncompressed disk space used by TrendMiner databases.

    database disk space