Jekyll2021-06-19T21:38:03+00:00https://aloisklink.com/feed.xmlAlois Klink’s WebsiteAlois Klink's Personal Website.Alois Klinkalois.klink@gmail.comGeotagging Photos using Google Location History and Exiftool2021-06-19T00:00:00+00:002021-06-19T00:00:00+00:00https://aloisklink.com/geotagging-photos-exiftool<p>A few years back, I went on a hiking trip, where I used a Campark ACT74
Action Camera, an extremely cheap 4K camera.</p>
<p>One of the features was that it could take a 4K picture 5 times a second,
or a 4K 60fps video in 5 minute chunks. Although the low battery life made
the camera difficult to use (and the fact that when the battery died), I made
almost 100 GB of photos/videos with this camera.</p>
<p>The camera didn’t have a GPS unit installed. Additionally, the camera had
an internal clock that constantly reset itself to 2017-01-01 whenever
the battery went out, so I also had loads of photographs incorrectly dated.</p>
<p>However, I realised that I could geotag these photos by using my Google Location
History and <code class="language-plaintext highlighter-rouge">exiftool</code>, so it was off to do that,
before <a href="https://www.theverge.com/2020/11/11/21560810/google-photos-unlimited-cap-free-uploads-15gb-ending">Google Photos ended their unlimited storage on June 1st 2021</a>.</p>
<h2 id="mounting-nas-folder-locally">Mounting NAS folder locally</h2>
<p>Firstly, since my photos were on my NAS, and I wanted to view the photos and
videos locally, I used SSHFS to mount all the photos locally.</p>
<p>SSHFS is slower than other methods of mounting a folder, but it’s definitely
the easiest, so it’s what I used.</p>
<p>I disabled compression and used the fastest cipher still supported by my
OpenSSH server/client, since as my NAS was on my local LAN, I was not worried
about security or network speed, I was more worried about my slow NAS CPU!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> <span class="nt">--parents</span> /tmp/geotag-photos
sshfs server:/zfspool/geotag-photos /tmp/geotag-photos <span class="nt">-o</span> <span class="nv">Compression</span><span class="o">=</span>no <span class="nt">-o</span> <span class="nv">Ciphers</span><span class="o">=</span>aes128-ctr
</code></pre></div></div>
<h2 id="getting-geographical-information-from-google">Getting Geographical Information from Google</h2>
<p>To geotag my images, I used data from Google Takeout, both my Google Fit
workout history in Garmin TCX format, and my Google Location history, in JSON
format.</p>
<p>I exported both of these as a gzipped tar (<code class="language-plaintext highlighter-rouge">.tgz</code> file).</p>
<p>Then, to extract the files I wanted, I ran the following:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> <span class="nt">--parents</span> ../extracted/google-fit
<span class="nb">tar</span> <span class="nt">-xvf</span> takeout-20201212T143859Z-001.tgz <span class="nt">--directory</span><span class="o">=</span>../extracted/google-fit <span class="nt">--strip-components</span><span class="o">=</span>3 <span class="s1">'Takeout/Fit/Activities/'</span>
<span class="nb">tar</span> <span class="nt">-xvf</span> takeout-20201212T143859Z-001.tgz <span class="nt">--directory</span><span class="o">=</span>../extracted <span class="nt">--strip-components</span><span class="o">=</span>2 <span class="s1">'Takeout/Location History/Location History.json'</span>
</code></pre></div></div>
<h2 id="converting-google-location-histroy-json-to-kml">Converting Google Location Histroy JSON to KML</h2>
<p>Next, I wanted to use <a href="https://exiftool.org/geotag.html">exiftool to geotag my images</a>.</p>
<p>However, unfortunately, although exiftool currently supports the Garmin TCX format
produced by Google Fit, it doesn’t support the Google Histroy GeoJSON file.</p>
<p>Because of that, I used
<a href="https://github.com/Scarygami/location-history-json-converter">Scarygami/location-history-json-converter</a>
to convert the Google History GeoJSON file to a KML file that <code class="language-plaintext highlighter-rouge">exiftool</code> supports.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> ../extracted
git clone https://github.com/Scarygami/location-history-json-converter.git
python3 location-history-json-converter/location_history_json_converter.py <span class="s1">'Location History.json'</span> location-history.kml
</code></pre></div></div>
<p>Then, finally I copied over all the location history tracks to my NAS:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> <span class="nt">--parents</span> /tmp/geotag-photos/geoinfo
<span class="nb">cp</span> <span class="nt">-r</span> location-history.kml google-fit/ /tmp/geotag-photos/geoinfo
</code></pre></div></div>
<h2 id="geotagging-images">Geotagging images</h2>
<p>Firstly, I found that some of my images were correctly dated, but some of
my images were set to <code class="language-plaintext highlighter-rouge">2017-01-01</code>, probably due to the internal clock of the
camera getting accidentally reset.</p>
<h3 id="correctly-dated-images">Correctly dated images</h3>
<p>Firstly, I copied over all the correctly dated images and videos into a new
folder:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> <span class="nt">--parents</span> accurate-time/ungeotagged
<span class="nb">cp</span> ../camera/<span class="o">{</span>photo,video<span class="o">}</span>/201806<span class="k">*</span> accurate-time/ungeotagged
</code></pre></div></div>
<p>First, I checked what the timezone for <code class="language-plaintext highlighter-rouge">DateTimeOriginal</code> was on my images:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>me@server:/zfspool/geotag-photos <span class="nv">$ </span>exiftool <span class="nt">-DateTimeOriginal</span> accurate-time/ungeotagged/20180613_191228A.jpg
Date/Time Original : 2018:06:13 19:12:28
</code></pre></div></div>
<p>However, <code class="language-plaintext highlighter-rouge">DateTimeOriginal</code> did not exist on my videos,
and the one in my photos was missing the correct timezone, so I had to add that in
using their filename:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>exiftool <span class="s1">'-datetimeoriginal<${filename}+01:00'</span> <span class="nt">-overwrite_original</span> accurate-time/ungeotagged/
</code></pre></div></div>
<p>Looks like it’s in UTC+00:00 (the photos were taken in +01:00, aka British Summer Time).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> <span class="nt">--parents</span> <span class="s1">'accurate-time/geotagged'</span> <span class="c"># output directory</span>
exiftool <span class="nt">-geotag</span> <span class="s1">'geoinfo/google-fit/2018-06-1*.tcx'</span> <span class="nt">-geotag</span> <span class="s1">'geoinfo/location-history.kml'</span> <span class="s1">'-geotime<${DateTimeOriginal}+01:00'</span> <span class="nt">-o</span> <span class="s1">'accurate-time/geotagged/'</span> <span class="s1">'accurate-time/ungeotagged'</span>
</code></pre></div></div>
<p>For Google Photos to understand the geotagged location of the videos, we
<a href="https://exiftool.org/forum/index.php?topic=11040.0">additionally need to add the <code class="language-plaintext highlighter-rouge">Keys:GPSCoordinates</code> tag</a>.</p>
<p>As Google Photos doesn’t seem to like floating-point numbers with more than 8
decimal points, we can limit this by using <code class="language-plaintext highlighter-rouge">-coordFormat '%.8f'</code>.</p>
<p>We can do this via:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>exiftool <span class="nt">-coordFormat</span> <span class="s1">'%.8f'</span> <span class="s1">'-keys:GPSCoordinates<$GPSLatitude, $GPSLongitude'</span> <span class="nt">-overwrite_original</span> accurate-time/geotagged/<span class="k">*</span>.mp4
</code></pre></div></div>
<h3 id="incorrectly-dated-images">Incorrectly dated images</h3>
<h4 id="2018-06-12-afternoon">2018-06-12 Afternoon</h4>
<p>Firstly, I copied over the first batch of photos that were incorrectly dated
starting from: <code class="language-plaintext highlighter-rouge">2017-01-01</code>.</p>
<p>I manually looked through the images and found that the one called
<code class="language-plaintext highlighter-rouge">20170101_014459A.jpg</code> was roughly at <code class="language-plaintext highlighter-rouge">20180612-203800</code>.</p>
<p>This meant that the time offset was (running this in python):</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="n">labelled_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="n">fromisoformat</span><span class="p">(</span><span class="s">'2017-01-01T01:44:59'</span><span class="p">)</span>
<span class="n">actual_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="n">fromisoformat</span><span class="p">(</span><span class="s">'2018-06-12T20:38:00'</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">actual_date</span> <span class="o">-</span> <span class="n">labelled_date</span><span class="p">)</span>
<span class="c1"># 527 days, 18:53:01
</span></code></pre></div></div>
<p>To fix these, we can then run:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> <span class="nt">--parents</span> 2018-06-12-afternoon/incorrect-time
exiftool <span class="s1">'-AllDates<${filename}+01:00'</span> ./2018-06-12-time-fucked/afternoon/ <span class="nt">-o</span> ./2018-06-12-afternoon/incorrect-time/
exiftool <span class="s1">'-AllDates+=1:05:11 18:53:01'</span> ./2018-06-12-afternoon/incorrect-time/ <span class="nt">-o</span> ./2018-06-12-afternoon/ungeotagged/
exiftool <span class="nt">-geotag</span> <span class="s1">'geoinfo/google-fit/2018-06-1*.tcx'</span> <span class="nt">-geotag</span> <span class="s1">'geoinfo/location-history.kml'</span> <span class="s1">'-geotime<${DateTimeOriginal}+01:00'</span> <span class="nt">-o</span> <span class="s1">'2018-06-12-afternoon/geotagged/'</span> <span class="s1">'./2018-06-12-afternoon/ungeotagged/'</span>
exiftool <span class="nt">-coordFormat</span> <span class="s1">'%.8f'</span> <span class="s1">'-keys:GPSCoordinates<$GPSLatitude, $GPSLongitude'</span> <span class="nt">-overwrite_original</span> ./2018-06-12-afternoon/geotagged/<span class="k">*</span>.mp4
</code></pre></div></div>
<h4 id="2018-06-12-morning">2018-06-12 Morning</h4>
<p>I did the same thing with the next batch, taken on the same day, but in the
morning.</p>
<p>Luckily, I had a frame in the video where I took a picture of my phone,
and the time on it, so that I knew that <code class="language-plaintext highlighter-rouge">2017-01-01 03:00:00</code> was equal to
roughly <code class="language-plaintext highlighter-rouge">20180612-161100</code>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="n">labelled_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="n">fromisoformat</span><span class="p">(</span><span class="s">'2017-01-01T03:00:00'</span><span class="p">)</span>
<span class="n">actual_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="n">fromisoformat</span><span class="p">(</span><span class="s">'2018-06-12T16:11:00'</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">actual_date</span> <span class="o">-</span> <span class="n">labelled_date</span><span class="p">)</span>
<span class="c1"># 527 days, 13:11:00
</span></code></pre></div></div>
<p>To fix these, we can then run:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> <span class="nt">--parents</span> 2018-06-12-morning/incorrect-time
exiftool <span class="s1">'-AllDates<${filename}+01:00'</span> ./2018-06-12-time-fucked/morning/ <span class="nt">-o</span> ./2018-06-12-morning/incorrect-time/
exiftool <span class="s1">'-AllDates+=1:05:11 13:11:00'</span> ./2018-06-12-morning/incorrect-time/ <span class="nt">-o</span> ./2018-06-12-morning/ungeotagged/
exiftool <span class="nt">-geotag</span> <span class="s1">'geoinfo/google-fit/2018-06-1*.tcx'</span> <span class="nt">-geotag</span> <span class="s1">'geoinfo/location-history.kml'</span> <span class="s1">'-geotime<${DateTimeOriginal}+01:00'</span> <span class="nt">-o</span> <span class="s1">'2018-06-12-morning/geotagged/'</span> <span class="s1">'./2018-06-12-morning/ungeotagged/'</span>
exiftool <span class="nt">-coordFormat</span> <span class="s1">'%.8f'</span> <span class="s1">'-keys:GPSCoordinates<$GPSLatitude, $GPSLongitude'</span> <span class="nt">-overwrite_original</span> ./2018-06-12-morning/geotagged/<span class="k">*</span>.mp4
</code></pre></div></div>
<h4 id="2018-06-13">2018-06-13</h4>
<p>Luckily, one of my pictures was on my phone, so I had an exact time match:
<code class="language-plaintext highlighter-rouge">2017-01-01T15:41:32</code> was roughly equal to <code class="language-plaintext highlighter-rouge">2018-06-13T16:14:30</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="n">labelled_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="n">fromisoformat</span><span class="p">(</span><span class="s">'2017-01-01T15:41:32'</span><span class="p">)</span>
<span class="n">actual_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="n">fromisoformat</span><span class="p">(</span><span class="s">'2018-06-13T16:14:30'</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">actual_date</span> <span class="o">-</span> <span class="n">labelled_date</span><span class="p">)</span>
<span class="c1"># 528 days, 0:32:58
</span></code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> <span class="nt">--parents</span> 2018-06-13-afternoon/incorrect-time
exiftool <span class="s1">'-AllDates<${filename}+01:00'</span> ./unknown/20170101_1<span class="k">*</span> <span class="nt">-o</span> ./2018-06-13-afternoon/incorrect-time/
exiftool <span class="s1">'-AllDates+=1:05:12 0:32:58'</span> ./2018-06-13-afternoon/incorrect-time/ <span class="nt">-o</span> ./2018-06-13-afternoon/ungeotagged/
exiftool <span class="nt">-geotag</span> <span class="s1">'geoinfo/google-fit/2018-06-1*.tcx'</span> <span class="nt">-geotag</span> <span class="s1">'geoinfo/location-history.kml'</span> <span class="s1">'-geotime<${DateTimeOriginal}+01:00'</span> <span class="nt">-o</span> <span class="s1">'2018-06-13-afternoon/geotagged/'</span> <span class="s1">'./2018-06-13-afternoon/ungeotagged/'</span>
exiftool <span class="nt">-coordFormat</span> <span class="s1">'%.8f'</span> <span class="s1">'-keys:GPSCoordinates<$GPSLatitude, $GPSLongitude'</span> <span class="nt">-overwrite_original</span> ./2018-06-13-afternoon/geotagged/<span class="k">*</span>.mp4
</code></pre></div></div>
<h2 id="uploading-images-to-google-photos">Uploading Images to Google Photos</h2>
<p>Finally, in order to upload the photos/videos,
I just opened up <a href="https://photos.google.com">https://photos.google.com</a> in my local computer’s web-browser,
and dragged the <code class="language-plaintext highlighter-rouge">sshfs</code>-mounted NAS folder to the web-page.</p>
<p>Although my intranet uses very slow Ethernet-over-Powerline (<100 Mbps),
which makes <code class="language-plaintext highlighter-rouge">sshfs</code> very inefficient, my internet upload speed is very limited,
and is less than 20Mbps, which remains the bottle-neck.</p>
<h2 id="manually-correcting-video-times-on-google-photos">Manually correcting video times on Google Photos</h2>
<p>Unfortunately, once everything was uploaded, my videos still had the incorrect
time. Their UTC time was shown as the local time, which in Britian during the summer,
is +01:00 hour behind.</p>
<p>I didn’t bother to use <code class="language-plaintext highlighter-rouge">exiftool</code> to fix this, as I was too close to the deadline
to reupload many gigabytes of videos. So instead, I just manually selected all
the videos in the Google Photos browser, pressed <strong>Edit date & time</strong>,
then <strong>Shift dates & times</strong>, and manually gave all the videos a 01:00 hour offset.</p>
<p>It’s not the best way of doing this, I’d have to find the correct flag to label
videos for Google Photos using <code class="language-plaintext highlighter-rouge">exiftool</code> in the future, but with the lack
of unlimited storage for Google Photos nowadays, it’s unlikely I will upload
many more vidoe files.</p>Alois Klinkalois.klink@gmail.comA few years back, I went on a hiking trip, where I used a Campark ACT74 Action Camera, an extremely cheap 4K camera.Migrating to OpenZFS and Shucking HDDs2021-04-18T00:00:00+00:002021-04-18T00:00:00+00:00https://aloisklink.com/zfs-and-shucking<p>Migrating an old 3 HDD mdadm RAID-5 array to an 3 HDD OpenZFS RAIDZ-2 array,
when my NAS case only has space for 4 HDDs.</p>
<h2 id="background">Background</h2>
<p>My old NAS was finally starting to run out of space, both in terms of GBs,
and in terms of space for additional HDDs.</p>
<p>As my old HDDs were four aging 2 TB HDDs in a RAID5 configuration using <code class="language-plaintext highlighter-rouge">mdadm</code>,
I decided to move to newer, larger, HDDs.</p>
<p>Using <a href="https://diskprices.com/?locale=uk">Disk Prices (UK)</a> I found that
I could buy a “WD 14 TB Elements Desktop External Hard Drive - USB 3.0, Black”
for a pretty cheap price, only £210 or £15.00 per TB.</p>
<p>Although this was an external USB HDD, I was planning on “shucking” them, e.g.
removing the outer USB enclosure, and taking out the internal HDD, as the
prices were significantly cheaper, and I felt the savings were worth the risk
of invalidating any waranty the HDDs had.</p>
<p>Additionally, HDDs designed for USB use would probably be quieter than
enterprise-grade HDDs, important for me, since I have to live
with my NAS.</p>
<p>Although lower-end WD drives use shingled magnetic recording (SMR), which
are <a href="https://arstechnica.com/gadgets/2020/06/western-digitals-smr-disks-arent-great-but-theyre-not-garbage/">basically unusable for RAID/ZFS</a>,
this only affects the lower capacity HDDs from Western Digital, not their
14 TB version.</p>
<p>I decided to go for at two parity drives, due to the URE issue: with a
unrecoverable read error (URE) rate of 1 per 10^14 bits (1 per 12.5 TB),
there is a good chance of getting a URE if a parity drive fails. So in the case
of a parity drive failing, we need another parity drive to prevent UREs and data
loss when rebuilding the array.</p>
<p>Finally, I decided to get for OpenZFS for my RAID solution. I previously did not
go for it, mainly because it was previously impossible to expand RAID arrays
in OpenZFS, which is somthing I often did with <code class="language-plaintext highlighter-rouge">mdadm</code>.</p>
<p>However, it looks like this feature is very close to completion
(see <a href="https://github.com/openzfs/zfs/pull/8853">openzfs/zfs#8853</a>), so hopefully
it will be ready if I ever need to increase the size of my RAID array.</p>
<h2 id="shucking">Shucking</h2>
<p>I bought 3x 14 TB WD Elements External Hard Drives from Amazon UK.</p>
<p>I followed <a href="https://www.ifixit.com/Guide/How+to+Shuck+a+WD+Elements+External+Hard+Drive/137646">How to Shuck a WD Elements External Hard Drive</a>
from iFixit and shucked the drives, to find:</p>
<ul>
<li>3x WDC WD140EDFZ-11A0VA0 (WD White)</li>
</ul>
<p>I didn’t bother doing any sort of testing on them,
and they’re not WD Red drives (just WD White), but a
<a href="https://old.reddit.com/r/DataHoarder/comments/elels8/wd_my_book_14_tb_shucked_wd140edfz_us7sap140/">review from schildzilla on reddit</a>
looks pretty positive!</p>
<h2 id="moving-data-process">Moving data process</h2>
<p>Unfortunately, my current NAS case only had space for an additional 3.5” HDD.
Because of that, my plan was to:</p>
<ul>
<li>Put in one 3.5” HDD and create a OpenZFS RAID-Z2 setup, with two “fake drives”,
so the array is in a degraded state.</li>
<li>Copy the data over from my old RAID array to the new degraded RAID array.</li>
<li>Remove the old RAID array, and install my two new 3.5” HDD, then let the
array rebuild. In case on any errors/issues on the array rebuild, I can still
reinstall the old RAID array.</li>
</ul>
<h3 id="creating-the-initial-openzfs-array">Creating the initial OpenZFS array</h3>
<p>Firstly, I installed one shucked 3.5” HDD into the only available space left
in the case of my NAS.</p>
<p>After turning my NAS back on, I ran <code class="language-plaintext highlighter-rouge">sudo fdisk -l</code> and could not find the
installed drive.</p>
<p>The drive follows the SATA 3.3 specification, where the 3rd power pin disables
the power of the drive. However, my old power supply unit (PSU) predates SATA 3.3,
and was constantly sending 3.3V into this power pin.</p>
<p>I loosely followed the guide from
<a href="https://www.instructables.com/How-to-Fix-the-33V-Pin-Issue-in-White-Label-Disks-/">Access Random</a>,
cutting a piece of tape over this pin.</p>
<p>Since pins 1 and 2 seem to not be used as well (see
<a href="https://www.tomshardware.com/news/hdd-sata-power-disable-feature,36146.html">SATA 3.3 pin diagram</a>
), I just lazily cut a big piece of tap that covered pins 1 to 3.</p>
<p>After booting back up my NAS, the HDD showed up in the BIOS, and I could see
all 12.75 TiB of it’s 14 TB glory after running <code class="language-plaintext highlighter-rouge">sudo fdisk -l</code>:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">me@me:~$</span><span class="w"> </span><span class="nb">sudo </span>fdisk <span class="nt">-l</span>
<span class="go">Disk /dev/sdf: 12.75 TiB, 14000519643136 bytes, 27344764928 sectors
Disk model: WDC WD140EDFZ-11
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
</span></code></pre></div></div>
<p>Then, I made two sparse files to generate the RAID-Z2 array on:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> <span class="nt">-p</span> ~/openzfs-tmp/ <span class="o">&&</span> <span class="nb">cd</span> ~/openzfs-tmp/ <span class="o">&&</span> <span class="nb">truncate</span> <span class="nt">--size</span><span class="o">=</span>14000519643136 fake1.img fake2.img
</code></pre></div></div>
<p>I didn’t place them into <code class="language-plaintext highlighter-rouge">/tmp</code>, since I was worried about losing them when
my NAS rebooted. <code class="language-plaintext highlighter-rouge">truncate</code> would make the files sparse, so they wouldn’t
actually use up 14 TB of space on my small boot drive.</p>
<p>I then created the ZFS pool:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>zpool create <span class="nt">-f</span> zfspool raidz2 /dev/sdf ~/openzfs-tmp/fake1.img ~/openzfs-tmp/fake2.img
</code></pre></div></div>
<p>Then immediately offlined the two temp files, so that they wouldn’t use up my
precious SSD space:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>zpool offline zfspool ~/openzfs-tmp/fake1.img ~/openzfs-tmp/fake2.img
</code></pre></div></div>
<p>And to confirm, the ZFS pool is now working fine:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">me@me:~/openzfs-tmp$</span><span class="w"> </span>zpool status zfspool
<span class="go"> pool: zfspool
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zfspool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
sdf ONLINE 0 0 0
/home/me/openzfs-tmp/fake1.img OFFLINE 0 0 0
/home/me/openzfs-tmp/fake2.img OFFLINE 0 0 0
errors: No known data errors
</span></code></pre></div></div>
<h4 id="zfs-options">ZFS Options</h4>
<p>Finally, there’s some options that are usually worth changing from default:</p>
<p>It’s usually worth adding <code class="language-plaintext highlighter-rouge">lz4</code> compression to your pool, unless you
have a slow CPU and most of your data is already compressed (e.g. video).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>zfs <span class="nb">set </span><span class="nv">compression</span><span class="o">=</span>lz4 zfspool
</code></pre></div></div>
<p>Setting <code class="language-plaintext highlighter-rouge">xattr=sa</code> gives much greater performance on Linux, although
makes the ZFS array incompatible with some other OSes.
Enabling <code class="language-plaintext highlighter-rouge">dnodesize=auto</code> is also good for as well
(but prevents GRUB from booting this).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>zfs <span class="nb">set </span><span class="nv">xattr</span><span class="o">=</span>sa <span class="nv">dnodesize</span><span class="o">=</span>auto zfspool
</code></pre></div></div>
<p>Setting <code class="language-plaintext highlighter-rouge">relatime=on</code> uses Linux’s <code class="language-plaintext highlighter-rouge">relatime</code> instead of <code class="language-plaintext highlighter-rouge">atime</code>. This essentially
massively cuts down on the number of <code class="language-plaintext highlighter-rouge">atime</code> (access time) writes made
whenever a file is read, back to the <code class="language-plaintext highlighter-rouge">ext4</code> defaults.</p>
<p>Unfortunately, <code class="language-plaintext highlighter-rouge">lazytime</code> is still not supported by ZFS
(see (openzfs/zfs#9843)[https://github.com/openzfs/zfs/issues/9843]),
so we can’t use that yet.</p>
<p>Disabling <code class="language-plaintext highlighter-rouge">atime</code> is also an option if you don’t think you’ll need it.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>zfs <span class="nb">set </span><span class="nv">relatime</span><span class="o">=</span>on zfspool
</code></pre></div></div>
<h3 id="copying-over-the-data">Copying over the data</h3>
<p>Firstly, I converted my existing old <code class="language-plaintext highlighter-rouge">mdadm</code> RAID5 pool into read-only mode.</p>
<p>First, I rand <code class="language-plaintext highlighter-rouge">cat /proc/mdstat</code>, to find the name of my <code class="language-plaintext highlighter-rouge">mdadm</code> pool, which
was <code class="language-plaintext highlighter-rouge">md0</code>:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">me@me:~/openzfs-tmp$</span><span class="w"> </span><span class="nb">cat</span> /proc/mdstat
<span class="go">Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sda[4] sdb[0] sdd[3]
5860147200 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [U_UU]
bitmap: 10/15 pages [40KB], 65536KB chunk
</span><span class="gp">unused devices: <none></span><span class="w">
</span></code></pre></div></div>
<p>Next, I remounted it in readonly mode:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>mount <span class="nt">--options</span> remount,ro /dev/md0 /mnt/md0
</code></pre></div></div>
<p>Finally, I opened up a <code class="language-plaintext highlighter-rouge">tmux</code> session (so that my copy command wouldn’t close if
my SSH connection dropped), and used <code class="language-plaintext highlighter-rouge">rsync</code> to copy the data over.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rsync <span class="nt">--partial-dir</span><span class="o">=</span>.rsync-partial <span class="nt">--info</span><span class="o">=</span>progress2 <span class="nt">--archive</span> /mnt/md0/ /zfspool/
</code></pre></div></div>
<p>Finally, many hours later, it was done!</p>
<h3 id="adding-in-the-new-hard-drives">Adding in the new hard drives</h3>
<p>Firstly, I exported and imported the zpool array. This converted the pool
from using <code class="language-plaintext highlighter-rouge">/dev/sd*</code> (which might change when we unplug devices),
to using <code class="language-plaintext highlighter-rouge">/dev/disk/by-id/*</code>, which should be consistent, even when we unplug
and plug in HDDs.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>zpool <span class="nb">export </span>zfspool <span class="o">&&</span> <span class="nb">sudo </span>zpool import <span class="nt">-d</span> /dev/disk/by-id
</code></pre></div></div>
<p>Next, I removed some of the old RAID HDDs, and replaced them with my newly
shucked HDDs to be used for OpenZFS parity, then turned the computer back on.</p>
<p>I had to first run the following to find the original <code class="language-plaintext highlighter-rouge">zpool</code> array:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>zpool import <span class="nt">-d</span> /dev/disk/by-id zfspool
</code></pre></div></div>
<p>Next, I used <code class="language-plaintext highlighter-rouge">sudo fdisk -l</code> to identify the device name of my newly installed HDDs.</p>
<p>And finally, I replaced the old temporary files I made the zpool with, with
the new HDDs:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>zpool replace <span class="nt">-f</span> zfspool ~/openzfs-tmp/fake1.img /dev/sda
<span class="nb">sudo </span>zpool replace <span class="nt">-f</span> zfspool ~/openzfs-tmp/fake2.img /dev/sdb
</code></pre></div></div>
<p>We can see with <code class="language-plaintext highlighter-rouge">zpool status</code> that resilvering is happening, as well as an
ETA for when it should be done. Fingers crossed we don’t get an UREs when
resilvering, otherwise we’d have to reinstall our original RAID HDDs.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">e@me:~$</span><span class="w"> </span>zpool status
<span class="go"> pool: zfspool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Apr 5 18:25:47 2021
2.57T scanned at 41.1G/s, 17.4G issued at 278M/s, 14.9T total
11.6G resilvered, 0.11% done, 0 days 15:31:31 to go
config:
NAME STATE READ WRITE CKSUM
zfspool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
sdf ONLINE 0 0 0
replacing-1 DEGRADED 0 0 0
/home/me/openzfs-tmp/fake1.img OFFLINE 0 0 0
sda ONLINE 0 0 0 (resilvering)
replacing-2 DEGRADED 0 0 0
/home/me/openzfs-tmp/fake2.img OFFLINE 0 0 0
sdb ONLINE 0 0 0 (resilvering)
errors: No known data errors
</span></code></pre></div></div>
<h4 id="debugging-infinite-loop-resilvering">Debugging Infinite Loop Resilvering</h4>
<p>A day or two later, I notice that one of my HDDs seems to be stuck in a resilvering loop.
It keeps on reaching 100%, then restarting at 0% again.</p>
<p>Time to test whether or not the HDD was damaged in shipping!</p>
<p>First, I ran <code class="language-plaintext highlighter-rouge">sudo zpool detach zfspool /dev/sdb</code> to cancel resilvering.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">me@me:~$</span><span class="w"> </span><span class="nb">sudo </span>zpool detach zfspool /dev/sdb
<span class="gp">me@me:~$</span><span class="w"> </span><span class="nb">sudo </span>zpool status
<span class="go"> pool: zfspool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Apr 6 17:27:57 2021
1.99T scanned at 5.46G/s, 159G issued at 435M/s, 14.9T total
44.0G resilvered, 1.05% done, 0 days 09:50:04 to go
config:
NAME STATE READ WRITE CKSUM
zfspool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
sdf ONLINE 0 0 2
sda ONLINE 0 0 2
/home/alois/openzfs-tmp/fake2.img OFFLINE 0 0 0
errors: 1 data errors, use '-v' for a list
</span></code></pre></div></div>
<p>Then, I tried doing a SMART test to quickly check if it could detect any issues
on the HDD, and found no issues with a short test:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>smartctl <span class="nt">--test</span><span class="o">=</span>short /dev/sdb
<span class="c"># wait a few minutes</span>
<span class="nb">sudo </span>smartctl <span class="nt">--log</span><span class="o">=</span>selftest /dev/sdb
</code></pre></div></div>
<p>I read on <a href="https://github.com/openzfs/zfs/issues/9551#issuecomment-633298283">openzfs/zfs#9551</a>
that the issue might be with me trying to resilver two devices at once,
so I redid my <code class="language-plaintext highlighter-rouge">zpool replace</code>
and crossed my fingers.</p>
<p>A day later, still no luck. Some more debugging later, looking at the <code class="language-plaintext highlighter-rouge">zpool</code>
events, we see that immediately when a resilvering finished, it restarted:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>zpool events <span class="nt">-v</span>
</code></pre></div></div>
<p>However, looking at the syslog showed an error with <code class="language-plaintext highlighter-rouge">zed</code>, it looks like
there is an error whenever resilvering finishes in the <code class="language-plaintext highlighter-rouge">resilver_finish-notify.sh</code>
script. Could this be the cause of the loop restarting?</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">me@me:~$</span><span class="w"> </span>vim /var/log/syslog
<span class="go">Apr 7 05:02:59 me zed: eid=56 class=history_event pool_guid=0x89A121A4820A4261
Apr 7 05:03:00 me zed: eid=57 class=resilver_finish pool_guid=0x89A121A4820A4261
Apr 7 05:03:00 me zed: error: resilver_finish-notify.sh: eid=57: "mail" not installed
Apr 7 05:03:00 me zed: eid=58 class=history_event pool_guid=0x89A121A4820A4261
Apr 7 05:03:04 me zed: eid=59 class=resilver_start pool_guid=0x89A121A4820A4261
Apr 7 05:03:05 me zed: eid=60 class=history_event pool_guid=0x89A121A4820A4261
</span></code></pre></div></div>
<p>Looking at the bash scripts, I found the following function in
<a href="https://github.com/openzfs/zfs/blob/master/cmd/zed/zed.d/zed-functions.sh"><code class="language-plaintext highlighter-rouge">zed-functions.sh</code></a>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>zed_notify_email<span class="o">()</span>
<span class="o">{</span>
<span class="c"># ...</span>
<span class="o">[</span> <span class="nt">-n</span> <span class="s2">"</span><span class="k">${</span><span class="nv">ZED_EMAIL_ADDR</span><span class="k">}</span><span class="s2">"</span> <span class="o">]</span> <span class="o">||</span> <span class="k">return </span>2
<span class="c"># ...</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Hang on, I’ve never set <code class="language-plaintext highlighter-rouge">ZED_EMAIL_ADDR</code>, why isn’t it returning early?
Look at <code class="language-plaintext highlighter-rouge">zed</code>’s config I can see the problem, <code class="language-plaintext highlighter-rouge">ZED_EMAIL_ADDR</code> was somehow
enabled. I uncommited it and waiting for the next resilverling loop to finish.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">me@me:~$</span><span class="w"> </span><span class="nb">sudo </span>vim /etc/zfs/zed.d/zed.rc
<span class="gp">#</span><span class="w"> </span>Email will only be sent <span class="k">if </span>ZED_EMAIL_ADDR is defined.
<span class="gp">#</span><span class="w"> </span>Disabled by default<span class="p">;</span> uncomment to enable.
<span class="gp">#</span><span class="w">
</span><span class="go">ZED_EMAIL_ADDR="root"
</span></code></pre></div></div>
<p>Still no luck (but it did fix the error message):</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">me@me:~$</span><span class="w"> </span>vim /var/log/syslog
<span class="go">Apr 7 16:42:06 elementalfrog zed: eid=68 class=history_event pool_guid=0x89A121A4820A4261
Apr 7 16:42:06 elementalfrog zed: eid=69 class=resilver_finish pool_guid=0x89A121A4820A4261
Apr 7 16:42:06 elementalfrog zed: eid=70 class=history_event pool_guid=0x89A121A4820A4261
Apr 7 16:42:11 elementalfrog zed: eid=71 class=resilver_start pool_guid=0x89A121A4820A4261
Apr 7 16:42:11 elementalfrog zed: eid=72 class=history_event pool_guid=0x89A121A4820A4261
</span></code></pre></div></div>
<p>I tried deleteing the file that was causing the error in <code class="language-plaintext highlighter-rouge">sudo zpool status -v</code>,
since I could just copy it from my old RAID array, but I still had no luck at the
end of the resilver.</p>
<p>Finally, I tried running <code class="language-plaintext highlighter-rouge">zpool detach zfspool /dev/sdb && zpool replace</code> again.
Then, almost 5 days after I started building the array, resilvering was finally
done.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> scan: resilvered 4.89T in 0 days 11:30:32 with 0 errors on Fri Apr 9 01:48:17 2021
config:
NAME STATE READ WRITE CKSUM
zfspool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
sdf ONLINE 0 0 8
sda ONLINE 0 0 8
sdb ONLINE 0 0 0
errors: No known data errors
</code></pre></div></div>
<p>However, I was still missing the final file that had the checksum error and
that I therefore deleted.
So I turned off the server, unplugged two of the OpenZFS drives again
(making a note that the action of unplugging them took off the tape
blocking PIN 3 of the SATA),
and plugged back in the <code class="language-plaintext highlighter-rouge">mdadm</code> RAID drives.</p>
<p>Finally, I first ran the rsync command with the <code class="language-plaintext highlighter-rouge">--dry-run</code> and <code class="language-plaintext highlighter-rouge">--verbose</code>
flags to see that the only file that would be copied would be the one I manually deleted.</p>
<h4 id="final-resilver">Final resilver</h4>
<p>After turning off the NAS, unplugging the old <code class="language-plaintext highlighter-rouge">mdadm</code> RAID devices, and plugging
in the OpenZFS devices (after taping up the SATA 3.3 pin 3), there was one final
resilver todo.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">me@me:~$</span><span class="w"> </span>zpool status
<span class="go"> pool: zfspool
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: resilvered 4.89T in 0 days 11:30:32 with 0 errors on Fri Apr 9 01:48:17 2021
config:
NAME STATE READ WRITE CKSUM
zfspool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
12345678900000000987 FAULTED 0 0 0 was /dev/sdf1
sda ONLINE 0 0 0
12345678900000000123 FAULTED 0 0 0 was /dev/sdb1
</span></code></pre></div></div>
<p>The 2 HDDs that I just added in showed as faulted, since they went offline.
To fix this, I just <code class="language-plaintext highlighter-rouge">export</code>-ed and re-<code class="language-plaintext highlighter-rouge">import</code>-ed them:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>zpool <span class="nb">export </span>zfspool <span class="o">&&</span> <span class="nb">sudo </span>zpool import <span class="nt">-d</span> /dev/disk/by-id zfspool
</code></pre></div></div>
<p>And that was the final resilver (relatively fast)!</p>
<p>Finally, to remove the ugly error logs, now that every is working,
I ran:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>zpool clear zfspool
</code></pre></div></div>
<h3 id="more-zfs-setup">More ZFS Setup</h3>
<p>Firstly, I split up my <code class="language-plaintext highlighter-rouge">zfspool</code> into seperate datasets.
This was so I could use different sharing mechanisms for each dataset,
as well as modify their block sizes for their specific use cases.</p>
<p>First, I created the <code class="language-plaintext highlighter-rouge">shared</code> folder, which I would use to store things shared
on the drive.</p>
<p>Next, I created a <code class="language-plaintext highlighter-rouge">shared/Media</code> folder, which I would increase the blocksize
for, for slightly increased performance, due to lower fragmentation.</p>
<p>I also disabled <code class="language-plaintext highlighter-rouge">atime</code> on some of the datasets, since they didn’t need it,
so I could reduce the uncessary writes.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>zfs create zfspool/shared <span class="o">&&</span> <span class="nb">sudo </span>zfs create zfspool/shared/Media
</code></pre></div></div>
<p>See the OpenZFS documentation on
<a href="https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#bit-torrent">Performance and Tuning | Workload Tuning</a>
for more information. For example, if you are using a database, you normally
want to decrease the blocksize to match your application.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>zfs <span class="nb">set </span><span class="nv">recordsize</span><span class="o">=</span>1M zfspool/shared/Media
</code></pre></div></div>
<p>Finally, once again, I had to use <code class="language-plaintext highlighter-rouge">rsync</code> to move my data from the root ZFS
folder to their appropriate datasets. I used the <code class="language-plaintext highlighter-rouge">--remove-source-files</code> to
automatically delete the original files after the <code class="language-plaintext highlighter-rouge">rsync</code>, so that the %-used
on the drive remained low.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rsync <span class="nt">--remove-source-files</span> <span class="nt">--partial-dir</span><span class="o">=</span>.rsync-partial <span class="nt">--info</span><span class="o">=</span>progress2 <span class="nt">--archive</span> /zfspool/Media /zfspool/shared
</code></pre></div></div>
<h2 id="sharing-openzfs-with-nfs">Sharing OpenZFS with NFS</h2>
<p>The next step was to replace my old NFS-share with the new ZFS sharing system.</p>
<p>Previously, I used the <code class="language-plaintext highlighter-rouge">/etc/exports</code> file, but ZFS has their own sharing
system.</p>
<p>Firstly, I commented out the lines in <code class="language-plaintext highlighter-rouge">/etc/exports</code> with my old config, then
ran <code class="language-plaintext highlighter-rouge">sudo exportfs -ra</code> to remove the entries.</p>
<p>Then, I enabled NFS sharing on the <code class="language-plaintext highlighter-rouge">/zfs/shared</code> folder for anybody on my local network (<code class="language-plaintext highlighter-rouge">192.168.0.*</code>):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>zfs <span class="nb">set </span><span class="nv">sharenfs</span><span class="o">=</span><span class="s1">'no_subtree_check,crossmnt,rw=192.168.0.0/24'</span> zfspool/shared
</code></pre></div></div>
<p>Finally, on my clients, I edited their <code class="language-plaintext highlighter-rouge">/etc/fstab</code> to the following:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">user@client:~$</span><span class="w"> </span>vim /etc/fstab
<span class="go">me.local:/zfspool/shared /mnt/server nfs4 _netdev,auto,soft,intr 0 0
</span></code></pre></div></div>
<p>And I ran <code class="language-plaintext highlighter-rouge">sudo mount --all --verbose</code> to mount everything in the <code class="language-plaintext highlighter-rouge">fstab</code> file
and confirm it worked!</p>
<h2 id="final-cleanup">Final cleanup</h2>
<p>Finally, I removed the old RAID config.</p>
<p>First, I removed the mount points in <code class="language-plaintext highlighter-rouge">/etc/fstab</code>.</p>
<p>Next, I deleted the array from <code class="language-plaintext highlighter-rouge">/etc/mdadm/mdadm.conf</code>.</p>
<p>And finally, I uninstalled the <code class="language-plaintext highlighter-rouge">mdadm</code> progra, since I wasn’t using it for
anything else:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt remove mdadm
</code></pre></div></div>
<p>It’s finally over, after more than a week of work 🥳.</p>Alois Klinkalois.klink@gmail.comMigrating an old 3 HDD mdadm RAID-5 array to an 3 HDD OpenZFS RAIDZ-2 array, when my NAS case only has space for 4 HDDs.