GPU Server IDC Operations

You mentioned “understanding Zabbix monitoring” — so how do you use Zabbix to monitor the video memory usage of GPU servers? Explain the specific steps clearly

“Steps for monitoring GPU video memory usage with Zabbix:

1. Install the Zabbix Agent on the monitored machine and ensure the nvidia-smi command is executable;
2. Write a script on the agent side (e.g. gpu_memory.sh) that extracts the video memory usage value via nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits;
3. Add UserParameter=gpu.memory.used,/path/to/gpu_memory.sh to zabbix_agentd.conf;
4. Restart the Zabbix Agent to apply the configuration;
5. Create a monitoring item on the Zabbix Server side (key: gpu.memory.used, type: Zabbix client);
6. Set up a trigger (e.g. alert when usage > 90%); 7. Create a graph associated with this monitoring item to enable visual display.”

You said you “participated in racking equipment in the IDC data center” — so what checks need to be done before racking a server? What are the operational standards during racking?

“Checks before racking a server:

• 1. Visual inspection (no dents on the chassis, no damaged ports, no loose screws);
• 2. Asset verification (SN number and configuration parameters match the work order);
• 3. Functional pre-testing (BMC management port pings successfully, hard disks / memory verified via pre-boot diagnostic tools, power module tested);
• 4. Rack compatibility check (server dimensions match the rack U position, PDU power meets the device’s power requirements, rail model is compatible).

Racking operation standards

• 1. Two-person collaboration (one person holds the server level while the other guides it into the rails; carrying by a single hand or dragging is prohibited);
• 2. Securing standards (tighten the server to the rails diagonally with screws of the specified specification, with torque meeting requirements);
• 3. Cabling requirements (separate power and data cables into different cable trays, cable labels facing the same direction, no blockage of server ventilation openings, redundant cables bundled neatly);
• 4. Power standards (dual power supplies connected to different PDU circuits to ensure redundancy);
• 5. Post-racking verification (power on and check the indicator lights, confirm hardware is recognized correctly via BMC, register assets in the CMDB system).

For a playbook that uses Ansible to upgrade NVIDIA drivers in batch, what are the core modules and tasks? At least 3 key tasks

Core modules

• apt / dnf (package management)
• systemd (service management)
• command / shell (run driver verification)

Key tasks (3)

1. Stop graphics / GPU-related services and uninstall the old driver
2. Install the specified NVIDIA driver version and DKMS dependencies
3. Load the nvidia kernel module and verify nvidia-smi works

Standard answer: “The core modules of an Ansible playbook for batch-upgrading NVIDIA drivers include yum/apt (dependency installation), copy (transferring the driver package), command (executing the driver installation command), systemd (managing services).

Key tasks:

1. Disable the nouveau driver (use the lineinfile module to modify the blacklist configuration, and the command module to run dracut -f to update initramfs);
2. Uninstall the old driver (use the command module to run nvidia-uninstall --silent);
3. 3. Install dependencies (use the yum module to install gcc, kernel-devel, dkms);
4. 4. Install the new driver (use the command module to run sh NVIDIA-Linux-x86_64-xxx.xx.run --silent);
5. 5. Verify the driver (use the command module to run nvidia-smi and check that the driver version in the output is correct).”

The job requirements mention needing to “handle simple sudden failures” — if a rack in the IDC data center suddenly loses power, what is your first reaction and what are your handling steps?

First reaction

Immediately determine whether it is a single-rack circuit breaker trip or an upstream power distribution cabinet / mains power failure; do not blindly re-close the breaker.

Handling steps

1. First check the rack for any burning smell, short circuit, or arcing to rule out hardware faults
2. Check the status of the PDU and the row-head cabinet circuit breakers; only try re-closing after confirming there is no device short circuit
3. After power is restored, start the servers one by one to avoid another trip from a sudden large current
4. Check business services, BMC, and network connectivity, and report the fault and recovery status in parallel
5. Wrap-up: record the cause of the power loss, handling duration, and affected devices and services; submit a written report to the supervisor, and assist in analyzing the root cause of the trip (e.g. whether it was due to device overload, cable aging, etc.).

Your resume says you “understand common network protocols (TCP/IP, HTTP, DNS, etc.)” — so describe the specific process of the TCP three-way handshake, and why three handshakes are needed instead of two?

1. TCP three-way handshake process

1. First (SYN): The client sends a connection request to the server; the segment’s flag SYN=1 is set, and an initial sequence number ISN is generated; the client enters the SYN_SENT state.
2. Second (SYN+ACK): After receiving the request, the server acknowledges receipt (ACK=1, acknowledgement number = ISN+1), and also sends a connection request (SYN=1) with its own generated ISN; the server enters the SYN_RCVD state.
3. Third (ACK): The client receives the server’s acknowledgement and request, and sends an acknowledgement segment (ACK=1, acknowledgement number = server ISN+1); the client enters ESTABLISHED; after receiving the acknowledgement, the server also enters ESTABLISHED, and the connection is established.

2. Why three handshakes instead of two?

The core reason is to prevent an obsolete connection request segment from being received by the server, which would establish an erroneous connection.

If only two handshakes were used:

• If the client’s connection request lingers due to network delay (not lost) and arrives at the server late, the server would mistakenly establish a connection, while the client has already discarded the request, causing wasted resources and abnormal connections.
• Through bidirectional acknowledgement, the three-way handshake both establishes reliable sending and receiving capabilities on both sides and avoids erroneous connections caused by such stale, invalid requests.

Suppose a server you maintain suddenly shows sustained 100% CPU usage, and it cannot be restarted for the time being — how would you troubleshoot and mitigate this problem step by step?

• Use top to locate the process and PID with the highest usage
• View process details with ps -fp PID to determine whether it is an abnormal / malicious process
• Check threads / IO / network to confirm whether it is compute-intensive or an infinite loop
• If necessary, use kill or kill -STOP to pause the abnormal process to relieve the CPU
• Check logs / application status to locate an infinite loop or bug in the business logic
• Observe the load returning to normal, confirm the scope of business impact, and report it

Suppose you now need to troubleshoot a network fault on a server: you cannot ping the gateway, but the local NIC light is on and the network cable is plugged in tightly. What steps would you follow? From hardware to software, explain the operation and the basis for each judgment

• Check the NIC status: ip addr Check whether the NIC is UP and whether the IP is configured correctly
• Check routing: ip route Confirm that the default gateway exists and is correct
• Check ARP: arp -n or ip neigh Check whether the gateway MAC can be resolved; if not, Layer 2 connectivity is down
• Check port configuration: confirm the switch port is not shut down, the VLAN is not misconfigured, and speed/duplex negotiation is not abnormal
• Check the firewall: iptables -L / firewall-cmd --list-all Check whether ICMP is disabled or outbound traffic is blocked, since ping relies on icmp
• Test the link: connect another device to the same cable / port to determine whether it is a server problem or a cabling / switch problem

When you interned at China Unicom, you were responsible for inspecting network equipment — so when inspecting a server, how do you judge whether the server hardware has a fault? No fluff, just give the steps directly

Standard answer: server hardware inspection should follow the process of “remote monitoring first, then on-site inspection”:

1. First check hardware monitoring data through the IPMI/BMC system: including the health status of CPU / memory / hard disks, power module redundancy status, fan speed and temperature thresholds, and hard disk SMART information (bad sectors / offline disks);
2. Check system logs (e.g. dmesg) for hardware errors (such as PCI device anomalies or hard disk IO errors);
3. During on-site inspection, check the physical state of the chassis: whether the power indicator is normal, whether there are any abnormal noises / odors, whether the temperature at the ventilation openings is too high, and whether cable connections are loose.

Your resume says you “handled a fault in the high school campus network” and restored the whole network within 2 hours — so what is the first step you take when troubleshooting a network fault? Stop with the irrelevant talk

Standard answer: the first step in troubleshooting a whole-network outage is “confirming the scope of the fault impact + verifying outbound network connectivity”:

1. First use an end device (e.g. just grab any student computer) to ping the gateway and ping an external DNS server, to determine whether it is an internal network fault or an egress fault;
2. Then check the physical status of the core equipment in the machine room (whether the power and run lights are normal) to rule out a hardware crash;
3. Only last, log into the core switch to check whether port status, VLAN configuration, or link aggregation is abnormal.

Your resume says you are “familiar with Linux Shell scripting” — so have you written a Shell script to check server hard disk status in batch? What core commands would the script use? Don’t tell me you’ve never written one

Standard answer: a Shell script for batch-checking server hard disk status uses two categories of core commands:

1. Check the physical health status of the hard disk: use smartctl -a /dev/sdX (to get SMART information);
2. Check the RAID array status (if any): use the tool for the corresponding RAID card, e.g. LSI’s megacli64 -LDInfo -Lall -aALL, or HPE’s hpssacli controller all show config.

Another question: the job requirements include deploying GPU servers — do you know which default driver in the Linux system must be disabled before installing the NVIDIA driver? Give the full name, not an abbreviation

The nouveau driver needs to be disabled.

The steps are:

1. Edit /etc/modprobe.d/blacklist.conf and add blacklist nouveau and options nouveau modeset=0;
2. Rebuild the initramfs image (e.g. dracut -f or update-initramfs -u);
3. After rebooting the system, use lsmod | grep nouveau to confirm the driver is disabled.

Another question: when racking servers on site at the IDC, do you know what requirements there are for the installation order of servers in a rack? Don’t tell me you just install them any way you like

Standard answer: the core principles for racking servers are “safety first, cooling priority”:

1. Layer by weight: place heavy equipment at the bottom and light equipment at the top, and ensure a single rack’s load does not exceed the rack’s rated capacity;
2. Arrange by cooling: leave at least 1U of space between devices, with the front facing the cold aisle and the rear facing the hot aisle, to avoid blocking airflow;
3. Plan by cabling: secure the rack PDU power first, then install the devices, ensuring power and network cables run through the cable managers on both sides of the rack to avoid tangling.

Another question: when using Prometheus to monitor GPU servers, what exporter needs to be deployed? What core metrics does it collect? Don’t try to bluff your way through this

1. First, what is being monitored is the NVIDIA GPU, using NVIDIA’s official dcgm-exporter
2. The core GPU metrics collected by dcgm-exporter include: GPU utilization, video memory usage, video memory bandwidth, GPU temperature, power consumption, ECC error counts, etc.

Another question: using Ansible to install the NVIDIA driver on 100 GPU servers in batch — what core modules do you need to write? What are the steps? Stop rambling

The correct core modules and steps are:

1. Use the yum or apt module to install dependencies (such as gcc, kernel-devel);
2. Use the copy module to upload the NVIDIA driver .run file to the target servers;
3. Use the command or shell module to execute the driver installation command (adding parameters such as --silent for non-interactive installation);
4. Use the modprobe module to load the nvidia kernel module;
5. Use the shell module to run the nvidia-smi command to verify whether the driver was installed successfully;
6. If a reboot is needed, use the reboot module

One last question: in the IDC data center, a server suddenly experiences frequent power losses — what do you check first? If you can’t answer, just leave

Standard answer:

1. Rush to the UPS cabinet immediately and check whether the upstream UPS supplying power is alarming and the status of its indicator lights (whether the mains power is interrupted and whether the battery is discharging); checking the upstream UPS instantly reveals whether it is a single-device or system-wide risk
2. Check whether the main PDU input circuit breaker has tripped, to rule out faults in the power supply chain;
3. If the UPS is normal, then check the PDU port and power module of the individual server;
4. Throughout the process, notify the business team in parallel and prioritize protecting the business by switching to a standby machine, rather than waiting idly to find the cause.

Your resume says you “participated in Linux server system installation” — so when installing CentOS 7, how do you achieve automated partitioning via a kickstart file? Requirements: root partition 50G, swap partition 8G, and the remaining space for the /data partition. Write the key configuration section and don’t miss any parameters

# Clear old partitions, initialize disk 
clearpart --all --initlabel 
# Partition configuration 
part /boot --fstype=xfs --size=1024 
part swap --size=8192 
part / --fstype=xfs --size=51200 
part /data --fstype=xfs --grow

Next question: after installing CUDA on a GPU server, how do you verify that CUDA was installed successfully and can be used normally? You must state the specific commands and verification points. Don’t get it wrong — this is your last chance; if you can’t even answer this, you really have no chance

# 1. Check the CUDA version
nvcc -V

# 2. Check the GPU status and driver version
nvidia-smi

# 3. Run the official sample test
cd /usr/local/cuda/samples/1_Utilities/deviceQuery
make
./deviceQuery

# 4. Bandwidth test (optional)
cd ../bandwidthTest
make
./bandwidthTest

• nvidia-smi works but nvcc -V reports an error → the CUDA Toolkit is not installed correctly or the environment variable is not configured (PATH needs to include /usr/local/cuda/bin)
• deviceQuery fails → the driver and CUDA version do not match, or the GPU is not recognized
• Multi-GPU environment: check whether nvidia-smi lists all GPUs

Next question: in GPU server operations, when you find that GPU utilization has been too high for a long time and video memory is also almost full, how would you troubleshoot the cause from the software level? You must give at least three troubleshooting directions with specific operations. This time you must answer well — don’t drop the ball again

Troubleshooting direction 1: Locate the occupying process

# View the specific process usage
nvidia-smi pmon -s um -o T

# Or in detailed mode
nvidia-smi -q -d PIDS | grep -A 5 "Process ID"

# After getting the PID, check the process details
ps -ef | grep <PID>
lsof -p <PID> | head -20

Troubleshooting direction 2: Analyze video memory leaks

# Monitor video memory changes in real time (refresh every second)
watch -n 1 nvidia-smi

# Use torch in a Python program to check memory allocation
python -c "
import torch
print(f'Allocated: {torch.cuda.memory_allocated()/1024**3:.2f} GB')
print(f'Reserved:  {torch.cuda.memory_reserved()/1024**3:.2f} GB')
print(f'Max:       {torch.cuda.max_memory_allocated()/1024**3:.2f} GB')
"

# Enable memory analysis (for PyTorch)
export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:51200

Troubleshooting direction 3: Check task scheduling and concurrency

# Check whether multiple tasks are competing for a single GPU
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

# Check container/K8s resource limits
docker stats --no-stream | grep <container_id>
kubectl top pod <pod_name> --containers

# Check whether tasks are over-provisioned
cat /proc/<PID>/cgroup | grep gpu

One more practical question: while your team is maintaining GPU servers, you suddenly receive an alert that a machine’s GPU temperature has soared to 95°C, video memory usage is abnormal, and SSH connections keep dropping. Please explain the troubleshooting approach and remedial measures step by step

1. Keep the connection: log in out-of-band via IPMI/iLO to bypass SSH and confirm it is not a network problem

2. Check the scene: use nvidia-smi -q -d TEMPERATURE to confirm the real temperature, and check whether the fan speed has dropped to zero

3. Kill processes: use nvidia-smi pmon to locate the PID with high video memory usage, force-terminate with kill -9, prioritizing cooling

4. Find the root cause:

   • Fan failure → report it for hardware replacement
   • Heatsink blockage → clean out the dust
   • Program infinite loop → check the code and logs

5. Prevent recurrence: temporarily lower the frequency with nvidia-smi -pl 200 (limit power consumption) and migrate the business to a standby machine

In one sentence: Use out-of-band access to keep it alive, kill processes to cool it down, check the fan/heatsink/code, and lower the frequency and migrate to protect the business.

Suppose the GPU server cluster you are responsible for suddenly has a large batch of unresponsive processes; the monitoring panel shows GPU video memory usage at 0 but the node status is “busy”, and at the same time the machine room reports that the power indicator lights of some racks are flickering. Drawing on your previous operational experience, explain your troubleshooting logic step by step from hardware to software, along with the core command or operational basis for each step

1. Hardware: use IPMI to check the power logs with ipmitool sel list, and confirm PDU fluctuations in the machine room
2. Bus: use lspci | grep nvidia to verify whether the GPU has been dropped, and dmesg to look for PCIe errors
3. Driver: if nvidia-smi returns blank, reload with modprobe nvidia, or reinstall the driver
4. Processes: use ps aux | awk '$8~/D/' to catch processes stuck in the D (uninterruptible) state, and clean them up with kill -9

One last question: during an IDC data center inspection, you find that a rack’s temperature and humidity exceed the limits — what is your handling process? This time if you can’t answer, you’re out

1. Confirm the scope: confirm whether the sensor is abnormal first, then check whether it is a single-rack or whole-row overage, and look at the sensor data of adjacent racks
2. Locate the heat source: find which device in the rack has the highest temperature, and use ipmitool sensor list to check the server’s intake/exhaust temperatures
3. Emergency cooling: temporarily increase the cold-aisle AC airflow, open the rack blanking panels to improve airflow, and shut down or migrate the high-heat devices
4. Find the root cause: is the AC faulty? Has the cold/hot aisle containment failed? Is device density too high due to over-provisioning?
5. Record and report: file a work order with the facilities team and state the urgency. Keep monitoring until the values return to normal thresholds

In one sentence: Define the scope, find the heat source, cool it down urgently, find the root cause, file a work order, and watch for recovery.

Now answer this: in GPU server operations, if you discover via the command line that a GPU is under sustained high load (over 90%) but no actual tasks are running, what are the possible causes? How should you troubleshoot and handle it?

Possible causes

• Zombie processes: a task has exited but the driver has not released the GPU context
• Driver anomaly: the kernel module is stuck and the GPU is in a busy-wait state
• Hardware fault: an internal GPU error causes the SMs to remain occupied
• Mining trojan: a hidden process is consuming compute power

Troubleshooting steps

1. Check processes: use nvidia-smi pmon -s um and fuser -v /dev/nvidia* to confirm no PID is occupying the GPU, but nvidia-smi shows high load → zombie/driver problem
2. Check video memory: nvidia-smi shows 0 video memory usage but 90% utilization → typical stuck driver
3. Check logs: use dmesg | grep -i nvidia to look for Xid errors, check /var/log/nvidia-installer.log for driver errors cps aux | grep -E "miner|xmr|pool", and use netstat -tulnp to look for abnormal external connections
4. Reset the GPU: use nvidia-smi -r -i <GPU_ID> for a hot reset, or rmmod nvidia; modprobe nvidia to reload the driver
5. Hardware isolation: if the reset fails, use nvidia-smi drain -p <PCI_BUS> to mark it unavailable, then migrate the business and replace the card

In one sentence: High load with no processes = zombie/driver/hardware/trojan; check the PID and video memory first, then scan logs and hidden processes, and replace the hardware if the reset fails.

Your resume says you “master RHCSA-level core skills” — so when configuring iptables rules in RHCSA, how do you restrict SSH access from a specific IP? Don’t just recite commands to me; explain how you actually verify that the rules have taken effect in real operations!

Configuration approach: deny by default, and only allow whitelisted IPs.

Methods to verify that the rules have taken effect:

1. Local verification: use iptables -L -n -v to check whether the rules are loaded and whether the counters show traffic (pkts/bytes non-zero means the rules were hit)
2. Peer-side test: SSH connections from an allowed IP should succeed, while connections from a denied IP should time out or be refused (Connection refused/timeout)
3. Packet capture confirmation: use tcpdump -i eth0 port 22 to see whether requests from the denied IP reach the local machine but are dropped
4. Log tracking: enable iptables -A INPUT -j LOG and check /var/log/messages for recorded DROP behavior
5. Persistence verification: run service iptables save, then restart the service and confirm that the rules persist and are not lost