GFS2 Puns

TLDR: Does GFS performance suck for use cases with many writers?

Inherited several RHEL{6,7} pacemaker/pcs clusters with 3-5 nodes. Configuration of each node is roughly:

• Four-socket, high core count, lower frequency CPUs with HT on (say 50+ physical cores and 2GHz)

• 1TB of RAM

• A single 10GbE interface that transports all incoming and outgoing data.

• Dual-port (A/B fabric setup) FC8 HBA

• HBA has access to four 10TB LUNs for GFS.

• Collectively, the four LUNs are composed of 75+ 10k rpm SAS spindles.

The purpose of these clusters is to receive many streams of data, write to GFS, do little-to-no processing, read from GFS, and send the output to other systems. Retention period for data they receive is short, a few days.

I know the configuration is, to be very kind, unbalanced. This is obvious  Each node has a 1.25GB/s network bottleneck from the single 10GbE interface, and a 1GB/s storage bottleneck, but 1TB of RAM and 50+ physical cores behind it to sit idle.

Each cluster has a few GFS2 filesystems utilizing the 40TB of "storage pool" (I'm not very familiar with GFS), the largest and most important is 30TB+ and doesn't hit over 20% capacity utilization.

GFS performance under a production load is very, very poor. Have ran simple 4-512MB dd tests that report less than 1MB/s performance to the big GFS file system, with iostat -xz running in a separate terminal that doesn't indicate there is any storage bottleneck (low dm utilization, await, service times), but there's many state 'D' processes. The "ping_pong" tool (from a Samba package, it does a loop of lock, tiny or null write, a read of that, then unlocks) test for file locking during production reports locks/sec below 100. I'm pretty certain that the state D processes are hung on actual file IO, and not IPC or... something.

Does GFS just suck for performance? Is its killerapp just HA? Can I make GFS work here? If this use case of GFS is abuse, what would you recommend as an alternative? A normal/posix filesystem to the application is required. I'm familiar with Lustre, but it's convenient to have compute and storage combined... if it worked. Despite hundreds of thousands of dollars invested in this architecture, there's absolutely no funding for an external (high performance) file system like Lustre or GPFS/SpectrumScale).

Any thoughts?

This may seem simple to some, but I'm looking for data points (official RHEL or otherwise) that point to why GFS2 performance would crawl on a 30TB filesystem (yes, 30TB, not GB) and not crawl on a 10TB filesystem.

On the smaller filesystem, we're seeing ~350mb/s write and on the larger one, we're seeing around 50-75mb/s write times. Trying to figure out why that is (even if it's just GFS2 and it's inability to crunch through as filesystems get larger)

What are some reasons we're seeing steep drops in transfer rates for both read and write?

Thanks

Normally I deploy oracle RAC on OEL. In this case I am unable to, so RHEL 7 it is. I would normally build this with ocfs2 as the target for rman backups and archived redo logs, but UEK 3 will not run on this HP hardware and I don't see an option for ocfs2 kernel support in RHEL 7. Has anyone used gfs or gfs2 for archived redo log and rman backup storage?

I have storage that I have shared block access to (multi-writer) but lack SCSI-3 persistent reservations. I was wondering if I could use other fencing (like the power switches) to handle this?

Hi, Pls help me, I have error below

[root@testweb01 ~]# pcs resource debug-start gfsvolfs_res Error performing operation: Operation not permitted Operation start for gfsvolfs_res:0 (ocf:heartbeat:Filesystem) returned 1 > stderr: INFO: Running start for /dev/sdb on /intellectapp > stderr: mount: mount /dev/sdb on /testapp failed: Transport endpoint is not connected > stderr: ocf-exit-reason:Couldn't mount filesystem /dev/sdb on /testapp

So I'm trying to configure a simple 2 node GFS2 cluster. They share a disk (/dev/sdc) that can be seen by both hosts just fine:

[root@msg4 by-id]# ls -lrt
total 0
lrwxrwxrwx 1 root root  9 Dec 22 12:48 ata-VMware_Virtual_IDE_CDROM_Drive_10000000000000000001 -> ../../sr0
lrwxrwxrwx 1 root root  9 Dec 22 12:48 lvm-pv-uuid-GGXKJf-2pHe-MI4G-R3zA-Jbwc-WfvR-0Gtkkr -> ../../sdb
lrwxrwxrwx 1 root root  9 Dec 22 12:48 wwn-0x600a09803830374c6d2b494857586533 -> ../../sdc
lrwxrwxrwx 1 root root  9 Dec 22 12:48 scsi-3600a09803830374c6d2b494857586533 -> ../../sdc

I created this gfs2 fs:

[root@msg5 ~]# mkfs.gfs2 -p lock_dlm -t cluster1:GFS -j 2  /dev/sdc
This will destroy any data on /dev/sdc.
It appears to contain: Linux GFS2 Filesystem (blocksize 4096, lockproto lock_dlm)

Are you sure you want to proceed? [y/n] y

Device:                    /dev/sdc
Blocksize:                 4096
Device Size                90.00 GB (23592960 blocks)
Filesystem Size:           90.00 GB (23592958 blocks)
Journals:                  2
Resource Groups:           360
Locking Protocol:          "lock_dlm"
Lock Table:                "cluster1:GFS"
UUID:                      72307a5b-0387-8446-2ca3-acf8aa450098

my cluster.conf:

<cluster config_version="8" name="cluster1">
  <fence_daemon/>
  <clusternodes>
    <clusternode name="msg5.testint.com" nodeid="1">
      <fence>
        <method name="pcmk-redirect">
          <device name="pcmk" port="msg5.testint.com"/>
        </method>
      </fence>
    </clusternode>
    <clusternode name="msg6.testint.com" nodeid="2">
      <fence>
        <method name="pcmk-redirect">
          <device name="pcmk" port="msg6.testint.com"/>
        </method>
      </fence>
    </clusternode>
  </clusternodes>
  <cman/>
  <fencedevices>
    <fencedevice agent="fence_pcmk" name="pcmk"/>
  </fencedevices>
  <rm>
    <failoverdomains/>
    <resources/>
  </rm>
</cluster>

Cman fires up okay:

[root@msg6 ~]# service cman start
Starting cluster:
   Checking if cluster has been disabled at boot...        [  OK  ]
   Checking Network Manager...                             [  OK  ]

I'm re-checking the current cluster-configuration which lead me to a few different questions.

There's a two node cluster with a network attached storage; everything with XFS, also the sharedstorage.

The sharedstorage should only be accessed by one node, but this can't be guaranteed (as they are also corrupting from time to time...)

I'm currently re-checking with RHEL Support and the Documentation, but would also be interested in what you guys use with a shared storage.

Any experiences/recommendations, or doesn't it matter anyway?

Thank you!

I'm working to create a shared apache environment with F5 working as the load balancer between 3 nodes running apache. My goal would be to utilize a clustered fs to only deploy/update a webapp once throughout all of the instances and also provide redundant hosts in the event of failure.

Anyone have any luck?

Hello,

I'm trying to figure out how to share a disk across 4 servers as a kind of virtual SAN. I have no access to a real SAN. From what I can tell I need to use GFS2 and distributed lock manager (DLM). I've setup a cluster but this is as far as I can get. I've been confused about fencing, STONITH (shoot the other node in the head) and references to DRDB (which seems unnecessary?).

Can someone tell me where I need to go next? Basically I want these machines to directly access their own LV and have that LV available with locking on the other three machines. The goal here is to have a virtual SAN that XenServer can use for VMs. I had it setup using GlusterFS before and it worked but taking one node out as a failover test blew things up too badly to recover. Long story but trust me it was terrible.

Here's my 'pcs status' output so far.

[root@hv1 ~]# pcs status
Cluster name: ha_cluster
Stack: corosync
Current DC: hv3storage (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum
Last updated: Thu Feb 23 20:15:16 2017		Last change: Thu Feb 23 20:14:26 2017 by root via cibadmin on hv1storage

4 nodes and 0 resources configured

Online: [ hv1storage hv2storage hv3storage hv4storage ]

No resources


Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Hi.

I need to set up GFS2 with two nodes, but the second node will only be available later this month. It's possible to setup a 1 node cluster with gfs2, and then add a second node later? Do i need to have the two nodes available at the very beginning?

Thanks for your time.

Looking for some expert opinions here.

First, a disclaimer. This post involves Oracle, but I'm not an Oracle guy, so I'll probably be misusing terminology here and there. Any appearance of me knowing what I'm talking about is simply me parroting what my Oracle guy tells me.

I have two RHEL7 servers that I want to set up as Oracle boxes in an active/passive configuration. This won't be a RAC: we'll be using whatever Oracle's version of log shipping is called in order to replicate data between the active and passive nodes.

In the past on RHEL6, I would accomplish that log shipping by having the active node write out its binlogs to a shared SAN volume set up with an OCFS2 filesystem mounted on both the active and passive node. Since RHEL7 no longer apparently supports OCFS2, I'm looking for an elegant solution.

Has anyone had any success doing something similar (or ideally identical) with GFS2? I've been doing a lot of reading this morning on the subject, but everything I've found so far makes the assumption that the GFS2 filesystem is being set up as part of a pacemaker/corosync cluster, which is something that I absolutely DON'T need in my scenario. If you have done something similar, would you be willing to share the steps required to get it working?

Thanks folks!

Go post NSFW jokes somewhere else. If I can't tell my kids this joke, then it is not a DAD JOKE.

If you feel it's appropriate to share NSFW jokes with your kids, that's on you. But a real, true dad joke should work for anyone's kid.

Mods... If you exist... Please, stop this madness. Rule #6 should simply not allow NSFW or (wtf) NSFL tags. Also, remember that MINORS browse this subreddit too? Why put that in rule #6, then allow NSFW???

Please consider changing rule #6. I love this sub, but the recent influx of NSFW tagged posts that get all the upvotes, just seem wrong when there are good solid DAD jokes being overlooked because of them.

Thank you,

A Dad.

Martin Freeman, and Andy Serkis.

They also play roles in Lord of the Rings.

I guess that makes them the Tolkien white guys.

'Eye-do'

This is my first post pls don't kill me lol.

The people in the comment section is why I love this subreddit!!

Cred once again my sis wants credit lol

I heard parents named their children lance a lot.

First post please don't kill me

Edit: i went to sleep and now my inbox is dead, thank you kind strangers for the awards!

second hand stores!

it's Hans free now..

Old Neeeeiiiiighvy

10+10 is twenty and 11+11 is twenty too

A buck-an-ear!

I Thank ye kind Matey for the booty! I be truly overwhelmed! Thank you!

Holy cow! Thank you everyone for the upvotes and awards! I wasn’t expecting this!

He should have a good vowel movement. His next diaper change could spell disaster though.

Making it all the way home and realizing that they forgot one of the containers:

Riceless

Without missing a beat he asks "Daddy, do you know how much room you need to grow Fungi like that?"

I did not know.

So he tells me "as Mushroom as possible!"

So proud.

Feyonce

He was so brave and even tried to encourage us, the family around him, with his last breaths. He kept whispering to us to Be Positive.

He said, “Change the batteries in your hearing aid”.

She was pretty mad when I only picked seven up