FreeNAS 11.2 and ESXi 6.7 iSCSI Tutorial

It’s been over 2 years since my previous guide on setting up iSCSI between FreeNAS and ESXi and in that time many things have changed. FreeNAS now has a new UI, making things simpler and more straight forward. I think we can all agree the prettier graphs are extremely important too.

In this guide we’ll evaluate if FreeNAS is still the best solution for your storage needs and explain why iSCSI performs best, followed by complete setup instructions for a killer multi-path redundant link iSCSI config. FreeNAS is great but as with most things, there are pros and cons so lets get them out of the way as clearly as possible.

Pros

  • Built on FreeBSD using ZFS file system, incredibly robust combination
  • ZFS uses lots of ram (instead of leaving it idle)
  • Copy on write file system, very resilient to power failures
  • Bit rot protection through check-summing and scrubs
  • ARC (adaptive replacement cache) gives excellent read performance, can be expanded to L2ARC with an SSD or Optane drive
  • ZIL is basically a fast persistent write cache and can also be put on an SSD or Optane Drive
  • lz4 compression is very fast and with VMs it’s not unusual to see 50% compression ratio

Cons

  • Requires a beefier system than traditional HW/SW raid solutions, 8-16GB ram is the low end
  • You can’t simply add/remove additional drives like you can with Unraid and MDADM

Is FreeNAS Right For You?

As of writing this in November 2019 I would lean towards Unraid for a pure file/media server. Due to it’s all in one nature, good VM support (running VMs on FreeNAS isn’t worth your time in my experience) and ability to add/remove disks it’s well suited for media/home use. However when performance starts to matter, FreeNAS shines with plenty of options to speed up VMs and/or databases and get great IOPS performance out of your spinning rust.

If you’re in the planning phase of your build, sticking to NAS hard drives for any setup is recommended. I recommend WD Red and Seagate Ironwolf drives as they’re rated for 24/7 use and are known to be highly reliable in NAS Servers. Going with cheaper drives may work however performance and reliability may be impacted. iXsystems (the creators of FreeNAS) have a range of systems available. Of course these are guaranteed to run FreeNAS excellent. You can browse their range here.

The Config

The goal is simple. We’ll be setting up 4 links between FreeNAS and ESXi (very easy to adapt this to 1-2 link setups). This supports failover and multipathing for a solid speed boost. I’ll show you some tricks to get better performance out of your setup along the way too!

The network I’ll be using for iSCSI is a simple 4 direct links between 2 servers. It’s best practice to have your iSCSI network separate from every other network.

FreeNAS      Cable   ESXi
10.0.0.1/30  <---->  10.0.0.2/30
10.0.0.5/30  <---->  10.0.0.6/30
10.0.0.9/30  <---->  10.0.0.10/30
10.0.0.13/30 <---->  10.0.0.14/30

FreeNAS Network

For the first step, I’m going to configure the IP addresses on my 4x connections I’ll be using for iSCSI. I’m using a /30 subnet meaning 2 hosts, perfect for this setup. You can configure it with any addresses you like as long as it’s reachable from ESXi and each link is on a different subnet.

Go to Network > Interfaces > ADD

Select your NIC, name it (I like to use the NIC as the name), set your IP and subnet mask. Repeat the ADD step for each link you’ll be using.

As a note, for additional performance you can add “mtu 9000” in the Options field. This will tell it to use Jumbo Frames and can result in higher throughput and lower CPU usage. However it can cause issues on some systems.

FreeNAS Storage

Next step I’m going to create a storage pool, if you’ve already done this skip this step.

Go to Storage > Pools and click the ADD button to create a pool.

Click CREATE POOL

Here you can name your pool. Then add the disks by selecting them and then using the right arrow to move them across. Select the raid type. In my case I’ll use mirror, however with more drives you can do raidz/raidz2 or a stripe of mirrors (raid10 like) for performance.

After you create your pool, click on the 3 dots next to it and select “Add Zvol”.

It’s recommended to not exceed 50% of your storage size for your iSCSI share so I’m going to make mine 115GiB

FreeNAS iSCSI

1. Time to setup iSCSI, we’ll first need to create a portal. To do this go to Sharing > Block (iSCSI) > Portals and click ADD.

Use the “ADD EXTRA PORTAL IP” option to allow you to add as many interfaces as you’ll need and type their IP Addresses in.

2. Next is Initiators > ADD. Here I’ve added a subnet that covers all my iSCSI networks, however you can leave it saying ALL and it’ll work A-OK for you.

3. Now on to Targets > ADD. We’ll fill in a name and set the portal group ID and initiator group ID to 1.

4. Extents > ADD. This is where we can pick our storage zvol under device. If you’re using HDDs you may like to select the LUN RPM however it’s not going to change anything I’m aware of.

5. Last step before we turn iSCSI on. Go to Associated Targets > ADD and select your target and extent from the lists. Set a LUN ID and save.

6. Go to Services and enable iSCSI, also checking the Start Automatically box.

ESXi Config

It’s time to configure ESXi, we have to setup the networking and attach the iSCSI storage to it.

ESXi Networking

To network this up to FreeNAS with the 4x links I’m using I’ll make 4 vSwitches. Go Networking > Virtual Switches > Add standard virtual switch.

Name your interface and pick the appropriate uplink. Note that if you use the “mtu 9000” option in your FreeNAS interface, you’ll have to set the mtu to 9000 here too.

We need to make 4 VMkernel NICs now. Click on VMkernel NICs > Add VMkernel NIC

Pick a name and select the appropriate vSwitch. Under IPv4 settings configure the static address and subnet to match the FreeNAS system. If you’re using mtu 9000, set that in the mtu aswell.

Repeat this for every interface and you’re good to go on the networking front.

ESXi iSCSI

iSCSI setup on ESXi is rather simple. Go Storage > Adaptors > Software iSCSI.

Enable iSCSI and fill in the 4 port bindings to the VMkernel NICs we created before. Then add dynamic targets to the FreeNAS IP Addresses.

Upon closing and opening the Software iSCSI configurator, you will see it picks up FreeNAS iSCSI share in the Static Targets area.

We have to add a datastore to finish this off. Go to Storage > Datastores > New datastore.

On the first page click next. It may take a minute, then you should see your FreeNAS zvol we created show up. Pick a name and click next.

Use the full disk with VMFS6 file system, go next and finish.

Guess what? It’s ready to test out!

iSCSI Testing and Tuning

In the current state you’ll notice it’s only getting single link speeds, be it 10gbit or 1gbit like I have.

To get higher speeds we need to set it to round robin and make it change it’s path every iop. By default it does it every 1000 iops.

To change this we need to go to the command line and input 3 commands. 3 commands isn’t that scary right?

Go to Manage > Services and click on TSM-SSH. Click Start above it to start the SSH service.

Open an SSH program like PuTTY and input your ESXi servers IP Address.

Click yes to the popup and login with your ESXi credentials that you use on the web interface. Type in the following.

esxcli storage nmp device list

looking at the output shown, find the FreeNAS iSCSI share and look for the naa ID numbers I’ve drawn around, taking note of them. Also note the bottom red box, we’ll change this setting.

Now paste in the following command, replacing the​ NAA_HERE with the contents of the red box starting with naa.. This will change the iSCSI mode to round robin

esxcli storage nmp device set --device NAA_HERE --psp VMW_PSP_RR

Following that command, put in this one to set the IOPs to 1.

for i in `esxcfg-scsidevs -c |awk '{print $1}' | grep NAA_HERE`; do esxcli storage nmp psp roundrobin deviceconfig set --type=iops --iops=1 --device=$i; done

After those 2 commands, type the very first one again to confirm it’s working. You should see something similar to the following.

After making these changes we can see that the performance is almost double. It’s far from 10gbe for me but with 2x 10gbit links you could achieve amazing results. Note that I’m using 2x 250GB Seagate HDDs from 2006 that are mirrored for this tutorial.

Share this post

Share on facebook
Share on google
Share on twitter
Share on linkedin
Share on pinterest
Share on print
Share on email

Join the Conversation

2 Comments

  1. Thank you for sharing your step by step guide on how to configure the FreeNas 11.2 along with ESX 6.7 for optimum performance.

    I don’t have access to dedicated hardware to build a FreeNas host and was thinking of using FreeNAS on a VM running on top of my lab ESX 6.7 host which does have a couple of 2x 10gbit NIC so I am thinking since the local network is running at 10gbit the performance should be reasonably good.

    Will be grateful if you are able to shed any light on the above.
    Thanks!

    1. Hi Manoj,
      As long as you pass through a PCI-E based SATA controller to the VM so it can see the hard drives directly it should work well.
      For iSCSI I believe running it on an internal network rather than over the NICs would be preferable then as there would be no speed cap. You could create a vSwitch with no NICs and put both management for ESXi (for iSCSI) and FreeNAS iSCSI on this vSwitch. I hope that makes sense and note, I haven’t tested it so I’m not 100% on it.
      Good luck and thank you!
      John

Leave a comment

Your email address will not be published. Required fields are marked *