I'm trying to get VPN access up and running. The company has a SonicWall firewall/concentrator and I'm working on a Mac. I'm not sure of the SonicWall's hardware or software level. My MacBook Pro is OS X 10.8, x64, fully patched.

The Mac Networking applet claims the remote server is not responding. The connection attempt subsequently fails:

This is utter garbage, as a Wireshark trace shows the Protected Mode negotiation, and then the fallback to Quick Mode:

I have two questions: (1) does Mac OS X VPN work in real life? (2) Are there any trustworthy (non-Apple) tools to test and diagnose the connection problem (Wireshark is a cannon and I have to interpret the results)?

And a third question (off topic): what is so broken in Cupertino such that so much broken software gets past their QA department? I pay good money for the software to run their hardware, and this is an absolute joke.

EDIT (12/14/2012, 6:00 PM): The network guy sent me "VPN Configuration Guide" (Equinox document SonicOS_Standard-6-EN). It seems an IPSec VPN now requires a Firewall Unique Identifier. Just to be sure, I revisited RFC 2409, where Main Mode, Aggressive Mode, and Quick Mode are discussed. I cannot find a reference to Firewall Unique Identifier.

EDIT (12/14/2012, 11:00 PM): From the Mac OS X logs (so much for the garbage message box from this crummy operating system):

Wed Nov 14 16:25:41 2012 : IPSec connection started

Wed Nov 14 16:25:41 2012 : IPSec phase 1 client started

Wed Nov 14 16:25:41 2012 : IPSec phase 1 server replied

Wed Nov 14 16:25:42 2012 : IPSec phase 2 started

Wed Nov 14 16:26:12 2012 : IPSec connection failed

...

Wed Nov 14 17:23:16 2012 : L2TP connecting to server '173.167.XXX.YYY' (173.167.XXX.YYY)...

Wed Nov 14 17:23:16 2012 : IPSec connection started

Wed Nov 14 17:23:16 2012 : IPSec phase 1 client started

Wed Nov 14 17:23:16 2012 : IPSec connection failed <IKE Error 23 (0x17) Invalid hash information>

EDIT (12/15/2012, 12:00 AM):

I think I am screwed here: http://forums.macrumors.com/showthread.php?t=383855. I am trying to connect to a broken (non-standard) firewall, with a broken Mac OS X client.

This thread comes up on a lot of Google searches for Mac OS X compatibility with SonicWall VPNs, so even though the thread is old, I just wanted to post that YES, Mac OS X's native VPN client works fine with SonicWall's L2TP VPN. Third-party VPN clients are nice and full-featured, but certainly not required. Proper configuration is necessary on the UTM-side, but the UTM admin should have confirmed Mac OS X compatibility before provisioning a VPN account to you (IMHO).

For troubleshooting, I recommend two things:

• 
  client-side: In the Mac OS X VPN's "Advanced" settings, enable "Use verbose logging" to get (much) more detailed output in the Console from the racoon client than you get by default.


• 
  UTM-side: While the Mac OS X racoon logs are very thorough, they can be hard to interpret and figure out what specific changes to make on the UTM or within the client. You're better off looking at the SonicWall "VPN xxxx" logs to figure out what is preventing the client from being able to connect. In particular, look for IKE mismatches errors.

That said, I have been successful setting up a L2TP VPN configuration that uses certificate authentication (not PSK) and IKEv2, and I can verify that it works for both native Mac OS X 10.10 and native Windows 7 VPN clients. Windows 8 should also be OK, but I cannot confirm. (IKEv1 with PSK auth also works, but I implore you to not configure the VPN this way -- it's not secure.). Please note that at no point is the use of the SonicWall's Firewall Unique Identifier needed.

I'm not an advocate of people blindly configuring settings that they don't understand. However, here are the settings I used in the hopes that admins who are new to setting up VPNs use the following as a template to stop using IKEv1 PSK, and properly configure a solid and secure VPN for their organization. The following is on a SonicWall NSA-series unit with SonicOS 5.8.x. I hope it helps:

1) VPN > Settings > VPN Policy > WAN GroupVPN settings

General tab

• 
  Authentication Method: IKE using 3rd Party Certificates


• 
  Gateway Certificate: {the cert you uploaded to the devce}
  


• 
  Peer ID Type: Distinguished name


• 
  Peer ID Filter: {the appropriate filter for your VPN client certs. The exact information you use to generate your client certs will determine the filter syntax you use.}
  


• 
  Allow Only Peer Certificates Signed by Gateway Issuer: {enable this if you are issuing your server and client certs all from the same CA (for example your own self-created CA)}
  

Proposals tab

• 
  DH Group: Group 2 (my testing determined that Group 5 was not supported by the native Mac OS X 10.10 or Windows 7 VPN clients)
  


• 
  Encryption: {choose based on overhead and speed. As of this writing, AES-128 is a decent tradeoff of security vs. speed.}
  


• 
  Authentication: SHA1 (not great, but slightly better than MD5)
  


• 
  Life Time: 28800

• 
  Protocol: ESP


• 
  Encryption: {maybe just match what you chose for Phase 1 to keep things simple}
  


• 
  Authentication: {match what you chose for Phase 1}
  

Advanced tab

(the Advanced Settings will change depending on your environment, so enable what you need)

• 
  Require authentication of VPN clients by XAUTH: enabled


• 
  XAUTH user group: {the name of the group you created for you VPN users}
  

Client tab

• 
  Cache XAUTH User Name and Password on Client: Session (chose Always if your users complain about needing to re-auth frequently)
  

• 
  Allow Connections to: This Gateway Only


• 
  Set Default Route as this Gateway: enabled (I wanted my clients to pass all traffic through the VPN)
  


• 
  Use Default Key for Simple Client Provisioning: disabled (we're using certs, so don't want this)
  

2) VPN > Advanced

• 
  Enable IKE Dead Peer Detection: enabled


• 
  Enable Fragmented Packet Handling: enabled


• 
  Enable NAT Traversal: enabled


• 
  Clean up active tunnels when Peer Gateway DNS name resolved to a different IP Address: enabled

• 
  Send IKEv2 Cookie Notify: enabled (we want IKEv2, since it's more secure)


• 
  IKEv2 Dynamic Client Proposal settings:
  {I was able to set DH Group 14/AES-256/SHA1 without problems with native Mac OS X 10.10 and Windows 7 clients. Choose what suits your environment}
  

3) VPN > L2TP Server

L2TP Server tab:

{set your DNS servers (and WINS servers, if needed)}

L2TP Users tab:

{Unless you are authenticating with RADIUS/LDAP, set your IP pool range and set the group to use as your VPN users}

PPP tab:

Set to this order: MSCHAPv2, CHAP, MSCHAP, PAP

SonicWall VPN does work with OSX devices, though not always out of the box. The SonicWall server's VPN policy has to be configured the right way. I've managed to get iPads and iPhones to connect to them once I got the VPN policy created right. If the SonicWall's VPN policy is not set up right, it just won't work.

I was able to connect OS X El Capitan to a Sonicwall TZ 215 using pre shared key (PSK), on the WAN GroupVPN. This was previously working for me with VPN Tracker, but now that I'm running El Capitan beta, VPN Tracker does not work, so I figured I'd give the native VPN another shot.

At first it wasn't working, and I thought I'd have to reconfigure the sonicwall as described by @AnnonymousCoward, to use certificates. However, I noticed in one of the KB documents referred to here that you should enable the Accept Multiple Proposals for Clients checkbox in the Advanced tab of the WAN GroupVPN if you're having problems connecting from iOS (and I figured, maybe OS X as well).

THIS WORKED.

To be clear, my WAN GroupVPN is configured for ESP: 3DES/HMAC SHA1 (IKE). Using Group2 for Phase 1. Life Time is 28800 on Phase 1 and 2. XAUTH is setup.

Under L2TP settings in the main VPN section of the Sonicwall, you must enable and configure the L2TP Server. I set mine up to assign IP addresses to trusted users (e.g. XAUTH users) in the same IP network range as the rest of my remote network.

On OS X side, I created a VPN (L2TP) connection. Server address is that of the remote firewall. Account name is that of the XAUTH user. Authentication settings has Password set as the XAUTH user password, and Shared Secret set as the PSK that was configured on Sonicwall. Group Name is left blank.

I haven't totally figured out routing. Normally in VPN tracker I define the network ranges that I want to route over the VPN (and they must match the routes that are defined on Sonicwall for the endpoint, e.g. 10.72.0.0/16 in my case). I can define multiple remote networks, if I need them, But I don't see where to specify that kind of setup in OS X's VPN configuration. However, so far I am not having a problem accessing the remote network. So I'm guessing L2TP works differently than the configuration I'm using in VPN Tracker.

IPSecuritas is free and it supports El Captain too.