When a Linux disk is full, you rarely find out through a friendly warning. You find out because a database refuses to start, a deployment aborts with “No space left on device”, SSH logins hang strangely or a website suddenly serves blank pages. The good news: the cause can almost always be found within minutes if you look in the right order. The bad news: there are at least four situations in which the usual commands show you something different from reality.
We don’t know this problem from books. At the end of September, the disk of our own server ran up to 100 percent, with zero bytes free. For this article we therefore didn’t just measure our production server, we also created small throwaway file systems on it and filled them on purpose. That let us reproduce each trap in isolation: deleted files that keep occupying space, a file system that reports “full” with 55 MB free, data that disappears under a mount point, and a config file that shrinks to zero bytes when saved.

If commands like df, du or find are still new to you, start with our Linux commands list. That covers the basic toolkit. This article is about the emergency: the disk is full, and you want to know why without deleting anything important along the way.
Linux disk full: the short version in seven steps
When you’re under pressure, you don’t need theory, you need an order of operations. This one has worked well for us:
- Which file system is full?
df -htells you. Don’t guess, because/,/varand/homecan be separate partitions. - Is it bytes or inodes?
df -i. If it says 100 percent there, deleting big files won’t help at all. - What is eating the space?
du -xh --max-depth=1 /path | sort -rh | head, or more comfortablyncdu -x /path. - Are there deleted but still open files?
lsof +L1. This is the most common explanation whendfanddudisagree. - Check the usual suspects:
journalctl --disk-usage,docker system df,/var/log, the package cache, old kernels, database binlogs, backups in the wrong place. - Clean up in a targeted way, with commands that only delete what they are supposed to.
- Make sure it doesn’t happen again: set limits for logs and a warning that reaches you before the disk is full.
Each step has its own pitfalls. We’ll go through them one by one, using what we actually measured.
What “No space left on device” really means
The error No space left on device (error code ENOSPC) sounds unambiguous, but it isn’t. The kernel returns it in several situations:
- There are no free data blocks left. That’s the classic case.
- There are no free inodes left. Every file needs exactly one inode, no matter how small it is. When they run out, you can’t create a new file even with gigabytes free.
- You’re a regular user, and the reserved blocks for root are the only thing still free.
- A quota is exhausted, or in containers the limit of the overlay file system.
The message is identical in all four cases. That’s why it’s pointless to start hunting for the biggest file at the first ENOSPC. First you need to know which resource is missing and on which file system.
Step 1: Use df to find out which file system is full
df -h
df (disk free) shows size, usage, free space and mount point for every mounted file system. On our server, at the time of writing, it looked like this:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 451G 400G 33G 93% /
Two things stand out that many people overlook. First, 451 minus 400 is 51, not 33. The missing 18 GB or so aren’t a rounding error, they are ext4’s reserved blocks (more on those shortly). Second, if you want to know which file system a particular path lives on, ask df about that path directly:
df -h /var/lib/mysql
findmnt -T /var/lib/mysql
That sounds trivial, but it prevents the most common cleanup mistake: you diligently delete things in /home while it’s actually /var, on its own partition, that is full. The space you free up ends up in the wrong account.
df -hT additionally shows the file system type. That matters because ext4, XFS and Btrfs behave differently when it comes to “full”. Btrfs, for example, often shows inaccurate values in df because metadata and snapshots are managed separately. There, btrfs filesystem usage / is the more reliable command.
Step 2: Check inodes with df -i
df -i
This command shows the same file systems, but counts inodes instead of bytes. On our server usage was a harmless 14 percent (just over 4 million of 29 million). It gets interesting when it says 100 percent.
Our test: full with 55 MB to spare
We created a small 64 MB ext4 file system with deliberately only 2,000 inodes and then wrote empty files until nothing worked anymore. After 1,988 empty files we got:
s/f1988: No space left on device
And now the decisive moment that causes confusion in every real incident:
$ df -h imnt
/dev/loop3 60M 64K 55M 1% /tmp/gmdisk/imnt
$ df -i imnt
/dev/loop3 2000 2000 0 100% /tmp/gmdisk/imnt
df -h reports 1 percent used and 55 MB free. Yet not a single new file can be created. Anyone who only looks at df -h will spend hours searching for a big file that doesn’t exist.

In practice, exhausted inodes are almost always caused by millions of tiny files: PHP sessions that never get cleaned up, framework cache directories, mail queues, thumbnails, or a script that creates temp files in a loop. You find the culprit by counting files, not size:
# Count files per directory, one level below /var
sudo find /var -xdev -type f | cut -d/ -f3 | sort | uniq -c | sort -rn | head
# Newer coreutils can do this directly
sudo du --inodes -x --max-depth=2 /var | sort -rn | head
Don’t delete them with rm -rf dir/*, because with millions of files the shell bails out with “Argument list too long”. Use find dir -type f -mmin +120 -delete instead: it only removes files older than two hours and doesn’t need an argument list.
Worth knowing for the future: on ext4, the number of inodes is fixed when the file system is created. The only way to get more later is to reformat. XFS and Btrfs allocate inodes dynamically, so the problem is much rarer there.
Step 3: Find the space hog with du and ncdu
Once df shows that it really is bytes, it’s time for du (disk usage). The most important switch is -x: it keeps du on the same file system so it doesn’t wander into other mounts like /proc, network shares or a second disk.
sudo du -xh --max-depth=1 / 2>/dev/null | sort -rh | head -10
On our server this single command took 37.7 seconds across the whole 400 GB disk, almost entirely system time spent walking directories. Result, shortened:
400G /
306G /root
34G /var
25G /usr
17G /opt
9.7G /tmp
5.0G /home
From there you work your way down: du -xh --max-depth=1 /root | sort -rh | head, then one level deeper, until you reach the culprit. That works, but it’s tedious, because every call counts from scratch.

ncdu: count once, then click through
That’s exactly what ncdu is for. It scans once and then gives you a sorted, navigable list in the terminal. Arrow keys move into directories, d deletes (after confirmation), q quits.
sudo apt install ncdu # Debian/Ubuntu
sudo dnf install ncdu # Fedora/RHEL
sudo ncdu -x /
A practical problem: ncdu wasn’t even installed on our server. On a truly full disk, the installation itself can fail because apt needs room for package lists and the package. In that case you’re left with du, or you first free up two or three obvious things (package cache, journal) and install it afterwards.
A tip for large servers: ncdu can export its scan to a file (ncdu -x -o /tmp/scan.ncdu /) and load it again later without rescanning (ncdu -f /tmp/scan.ncdu). The export file obviously shouldn’t live on the full partition.
If you prefer something more modern, gdu is a faster alternative with parallel scanning, and duf is a prettier take on df. Neither is required.
Searching for big files directly
Sometimes the problem isn’t a directory but a single huge file: a core dump, a forgotten backup, an ISO, a database export. You can find those directly:
sudo find / -xdev -type f -size +500M -printf '%s\t%p\n' 2>/dev/null | sort -rn | head
On our /var this search finished in 2.8 seconds, much faster than a full du, because find only checks metadata and doesn’t have to add anything up.
Trap: ls and du show different sizes
We created a file with truncate -s 1G. That produces a so-called sparse file: it has a logical size, but only occupies the blocks that were actually written to.
$ ls -lh sparse.img
-rw-r--r-- 1 root root 1.0G sparse.img
$ du -h sparse.img
0 sparse.img
$ du -h --apparent-size sparse.img
1.0G sparse.img
ls shows 1 GB, du shows 0. For the question “where did my space go?”, du is always the right answer, because it counts real blocks. VM disk images, loop-mounted container storage and some database files are often sparse. If you sort by ls -l while cleaning up, you’ll delete the wrong file.
Step 4: Deleted files that still take up space
This is the trap that costs the most time, and it’s surprisingly common in practice. The symptom: df says the disk is full. But du finds far less data. Gigabytes are missing somewhere, and no directory explains them.
Our test
On a 200 MB test file system we created a 100 MB file and started a process that kept it open. Then we deleted the file:
$ df -h mnt # before rm
/dev/loop3 172M 101M 58M 64% /tmp/gmdisk/mnt
$ rm mnt/big.log
$ df -h mnt # after rm
/dev/loop3 172M 101M 58M 64% /tmp/gmdisk/mnt
$ du -sh mnt
20K mnt
The file is gone, du only finds 20 KB, but df still reports 101 MB used. The reason: on Linux, rm only removes the directory entry. The data itself is released only once no process has the file open anymore. As long as anything still holds a file descriptor to it, the space stays allocated, invisible to du and ls.

The typical real-world trigger: someone spots a huge log file, deletes it with rm, and the service writing to it just carries on. It happily writes into a file that no longer exists in any directory, and the disk keeps filling up.
How to find such files
sudo lsof -nP +L1
+L1 means: show open files with fewer than one link, i.e. deleted ones. In our test there was exactly one line:
sleep 4082618 root 3r REG 7,3 104857600 0 12 /tmp/gmdisk/mnt/big.log (deleted)
You see the process name, PID, file descriptor (here 3), size (100 MB) and the old path marked (deleted). On our production server the same command returned 425 entries totalling about 48 MB. That’s harmless and normal: many programs deliberately work with deleted temp files. It becomes suspicious when a single entry is gigabytes in size.
If lsof isn’t installed, you can do without it:
sudo find /proc/*/fd -ls 2>/dev/null | grep '(deleted)'
How to get the space back
The cleanest fix is to restart the service (systemctl restart servicename) or tell it to reopen its log file (for many services via systemctl reload or a signal like SIGHUP or SIGUSR1). If a restart isn’t possible right now, you can empty the file through the process:
# PID and FD from the lsof output
sudo sh -c ': > /proc/4082618/fd/3'
In our test, usage immediately dropped from 64 to 1 percent without the process being terminated. That’s a stopgap: the process keeps writing to the (now empty) deleted file, so the problem slowly comes back until you restart the service properly.
The rule going forward: never delete a log file that’s in use with rm, truncate it instead:
sudo truncate -s 0 /var/log/myservice/huge.log
The file stays in place, the service keeps writing to it, and the space is freed instantly.
Step 5: Reserved blocks, or why root can still write
Remember the gap between 451 GB size, 400 GB used and only 33 GB free? Those are the reserved blocks. By default, ext4 reserves 5 percent of its blocks for root when the file system is created. In our 200 MB test system that was exactly 2,560 of 51,200 blocks according to tune2fs.
We tried it. First, we wrote as a regular user (nobody) until nothing more would fit:
dd: error writing 'mnt/fill': No space left on device
/dev/loop3 172M 158M 0 100% /tmp/gmdisk/mnt
df reported 100 percent and 0 free. After that, root was still able to write about 10 MB to the same file system before it, too, hit the wall. Usage went from 158 to 168 MB.
That’s intentional, not a bug. The reserve ensures that system services running as root, and you as the admin, can still act once regular users have filled the disk. Two practical consequences:
- If an application (running as its own user) reports “No space left” but root can still create files, the disk is still full for that application. The reserve is not a buffer for your services.
- On large data-only disks (for backups or media, for example) that don’t hold an operating system, 5 percent is a lot. On 4 TB that’s 200 GB. There you can lower the reserve:
sudo tune2fs -m 1 /dev/sdb1. Leave it alone on the system partition.
XFS doesn’t have this root reserve in the same form. Full there really means full, for everyone.
Trap: A full disk can wipe config files
This trap is mentioned in almost no guide, and it can turn an annoying problem into a real one. On the full test file system we created a small two-line config file and then tried to overwrite it with new content, exactly the way an editor, a script or a deployment tool would:
$ cat app.conf
port=8080
name=app
$ printf 'port=9090\nname=app\nfoo=bar\n' > app.conf
printf: I/O error
$ ls -l app.conf
-rw-rw-rw- 1 root root 0 app.conf
Afterwards the file is zero bytes. That’s down to how the > redirect works: the shell first opens the file for writing and truncates it to zero. Only then is the new content written, and that’s exactly the part that fails because there’s no space. The result is neither the old nor the new configuration, but nothing at all.
The same thing happens with programs that constantly rewrite their state file or configuration. If a service restarts with an empty configuration afterwards, it may come up with defaults, or not at all. So:
- Don’t overwrite anything on a full disk that you can’t afford to lose. Make space first, then make changes.
- In your own scripts, write files atomically: write the content to a temp file in the same directory, check that it worked, then rename it with
mv.mvreplaces the file in a single step, and the old version survives until the new one is complete.
tmp=$(mktemp ./app.conf.XXXX) && printf '...' > "$tmp" && mv "$tmp" app.conf
After a disk-full incident you should therefore not only free up space, but also check whether configurations or database files have ended up empty or corrupted. find /etc -xdev -type f -size 0 -newermt '-2 hours', for example, lists every file in /etc that changed in the last two hours and is now empty.
Trap: Data hidden under a mount point
The last case where df and du disagree is rare but particularly sneaky. On our test file system we created a directory data, wrote an 80 MB file into it, and then mounted a second file system right on top of that directory:
$ df -h mnt
/dev/loop3 172M 81M 78M 51% /tmp/gmdisk/mnt
$ du -shx mnt
20K mnt
81 MB used, but du only finds 20 KB. The file is still on disk, but hidden by the new mount. No ls and no du can see it while the mount is in place.
In real life this mostly happens when a drive wasn’t mounted at boot once, say a backup disk or a network share. A backup job or application then writes into the empty mount point directory on the system disk. At the next reboot the drive is back and covers the data. The system disk is fuller, and nobody knows why.
You can uncover this with a bind mount that shows the system partition again without the mounts layered on top:
sudo mkdir -p /mnt/rootview
sudo mount --bind / /mnt/rootview
sudo du -sh /mnt/rootview/mnt/* /mnt/rootview/media/* 2>/dev/null
sudo umount /mnt/rootview
In our test the bind mount showed the hidden 81 MB immediately. To prevent the problem, use the nofail option in /etc/fstab together with services that check whether their target really is a mount point (mountpoint -q /backup || exit 1) before they write.
The usual space hogs on a Linux server
If none of the traps apply, it’s almost always one of these candidates. We measured all of them on our own server.
The systemd journal
journalctl --disk-usage
Ours: 390.8 MB. That’s harmless because we set a limit in /etc/systemd/journald.conf (SystemMaxUse=1G). Without a limit, journald takes up to 10 percent of the file system, capped at 4 GB. On a small 20 GB VPS that’s quickly 2 GB just for logs. Cleaning up is instant and safe:
sudo journalctl --vacuum-size=500M # shrink to 500 MB
sudo journalctl --vacuum-time=14d # remove everything older than 14 days
To make it permanent, set the limit in the configuration and run sudo systemctl restart systemd-journald.
Docker: images, containers, volumes and logs
On many servers Docker is the single biggest consumer of space, and also the one where careless cleanup does the most damage. Look first:
docker system df
Ours:
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 15 6 16.16GB 4.551GB (28%)
Containers 8 8 1.065GB 0B (0%)
Local Volumes 4 4 2.103GB 0B (0%)
Build Cache 5 0 79.8MB 0B
4.5 GB could go, namely images no container uses anymore. There are graduated commands for that:
docker image prune # only "dangling" images (no tag)
docker image prune -a # all images without a running or stopped container
docker builder prune # build cache
Be careful with docker system prune --volumes. It additionally deletes every volume that isn’t attached to a container at that moment. If you stopped a database container to rebuild it, its data volume is “unused” right then, and gone. On our servers we never run a blanket prune with --volumes; we delete volumes only individually and by name.
An often overlooked space hog is container logs. The default json-file logging driver writes without any limit to /var/lib/docker/containers/<id>/<id>-json.log. A chatty container can fill many gigabytes over a few months. The fix in /etc/docker/daemon.json:
{
"log-driver": "json-file",
"log-opts": { "max-size": "50m", "max-file": "3" }
}
The setting only applies to newly created containers. Existing ones have to be recreated once (docker compose up -d --force-recreate). The basics are covered in our article What is Docker?.
Package cache and old kernels
du -sh /var/cache/apt # 620 MB on our server
sudo apt clean # empty the package cache
sudo apt autoremove --purge # packages no longer needed, including old kernels
apt clean is completely safe; packages are downloaded again when needed. Run apt autoremove without -y first and read the list. On Fedora and RHEL the equivalents are dnf clean all and dnf autoremove. On Ubuntu systems with Snap, /var/lib/snapd can also grow with old revisions. sudo snap set system refresh.retain=2 limits how many versions are kept.
If /boot is a small separate partition, that’s the one that likes to fill up, because old kernels pile up. Updates then fail with cryptic errors. autoremove fixes this in almost every case.
Log files in /var/log
On our server /var/log used just under 1 GB, which is unremarkable. It becomes a problem when a service writes its own logs that logrotate doesn’t cover, or when an error writes thousands of lines per second to a log in a loop. Then cleanup won’t help, only fixing the error will. A quick look at the largest logs:
sudo find /var/log -type f -size +100M -exec ls -lh {} \;
Rotated and compressed old logs (*.gz, *.1) can be deleted safely; the current file is emptied with truncate, as described above.
Databases: binlogs and WAL
With the binary log enabled, MySQL and MariaDB write every change to binlog files as well. Without an expiry, these grow indefinitely. Never delete them with rm, go through the database instead:
PURGE BINARY LOGS BEFORE NOW() - INTERVAL 7 DAY;
To make it permanent, set binlog_expire_logs_seconds (MySQL 8) or expire_logs_days (older MariaDB). How binlogs relate to backups and point-in-time recovery is something we measured in detail in our MySQL backup article. With PostgreSQL, it’s usually a stuck replication slot that holds on to WAL files. SELECT slot_name, active FROM pg_replication_slots; shows whether an inactive slot is to blame.
Backups in the wrong place
A classic we know from experience: a backup script writes an archive every night to the same disk it’s supposed to protect, and never deletes old copies. That’s doubly bad, because it fills the disk and also isn’t a real backup. A backup that dies together with the server protects against nothing. How to split it up properly is described in our 3-2-1 backup strategy.
Temp directories, core dumps and home directories
/tmp took up 9.7 GB on our server, mostly build directories and test data left behind after use. Many distributions only clean /tmp at boot, but servers are rarely rebooted. systemd-tmpfiles can remove old files automatically (configuration in /etc/tmpfiles.d/). Crashed programs also leave core dumps behind, which can be several gigabytes. coredumpctl list shows them; /var/lib/systemd/coredump/ is the usual location.
And then there are home directories. On our server, by far the biggest chunk, 306 GB, sat under /root, where projects, build artefacts and working copies live. That’s not a fault but working material, and it’s exactly why you shouldn’t delete blindly on a server.
The second video is in German, but the terminal demo of ncdu is easy to follow without understanding the narration.
Cleaning up safely: what you may delete and what you may not
The most expensive mistakes happen while cleaning up under time pressure. Our rules of thumb, from experience and sharpened again after the September incident:
Safe:
apt clean/dnf clean alljournalctl --vacuum-size=…- rotated logs (
*.gz,*.1,*.old) in/var/log docker image pruneanddocker builder prune- core dumps, if you don’t need them for debugging
Think first:
apt autoremove: read the list before confirmingdocker image prune -a: images will be pulled again on the next start, which costs time and bandwidth- contents of
/tmp: check first whether a process is currently using it (lsof +D /tmp/dir) - old backups: only if a newer, verified copy exists somewhere else
Don’t touch:
- files of running databases (
/var/lib/mysql,/var/lib/postgresql) directly withrm - Docker volumes via a blanket
prune --volumes - binlogs or WAL files with
rm - anything under
/usr,/lib,/boot(except through the package manager)
And regardless of all that: before you delete something you didn’t create yourself, ask whether a service depends on it. A directory being huge doesn’t automatically make it junk.

Emergency: the disk is 100 percent full and nothing works
When truly nothing works anymore, and even apt and editors refuse, this order helps:
- Stay calm and don’t overwrite anything. Don’t edit configurations, don’t reconfigure services (see the empty-file trap).
- Quickly free a few hundred MB:
journalctl --vacuum-size=200M,apt clean, delete one large rotated log. That gives you breathing room for everything else. - Check for deleted open files:
lsof +L1. Often the solution is right there. - Then search calmly with
duor the now installablencdu. - Afterwards, check what broke while the disk was full: empty files in
/etc, database logs for errors, services viasystemctl --failed.
If you can’t even get in via SSH because login itself needs write access, your hosting provider’s rescue console or a rescue system is what’s left. Mount the disk there and clean up from outside.
A trick many admins use preventively: a placeholder file, for example fallocate -l 2G /var/reserve.img. When the disk fills up, delete it and you instantly have 2 GB of breathing room for the rescue. Simple, but it permanently costs you 2 GB.
Making sure it doesn’t happen again
The real solution isn’t cleanup, it’s getting notified in time. Three things are enough on most servers:
1. Limits for everything that grows. journald with SystemMaxUse, Docker with max-size, databases with binlog expiry, your own logs via logrotate, backups with a fixed retention.
2. A warning at 80 to 85 percent. That can be a full monitoring stack, like the ones we compared in our server monitoring article, or a small script run by cron:
#!/bin/sh
# /usr/local/bin/disk-warn.sh
LIMIT=85
df -P -x tmpfs -x devtmpfs -x overlay | awk -v l=$LIMIT 'NR>1 {gsub("%","",$5); if ($5 >= l) print $6": "$5"%"}' | \
while read line; do
echo "Disk almost full on $(hostname): $line" | mail -s "Disk warning $(hostname)" admin@example.com
done
Make sure to monitor inodes too (df -iP in the same way), otherwise the warning will miss exactly the case from our test. How to run such a script on a schedule is covered in our guide create a cron job.
3. Look at the trend, not just the moment. A disk that grows by 1 GB every day is more dangerous at 70 percent than one that has sat at 90 percent for months. If you write the usage to a file once a day (df -h / >> /var/log/disk-trend.log), you’ll spot outliers long before they become a problem.
And finally, perhaps the most important rule: after every cleanup, check with df -h that the space has actually been freed. If it hasn’t, it’s almost always a deleted file that’s still open, or you cleaned up on the wrong file system.
Frequently asked questions about a full Linux disk
How do I check how full my disk is on Linux?
With df -h. It shows size, usage and free space for every mounted file system. For a specific directory use df -h /path, for inodes df -i.
Why is the disk full when du finds hardly any data?
Most often a running process is holding a file that has already been deleted. sudo lsof +L1 shows such files. Less often, data is hidden under a mount point, or the difference is ext4’s reserved blocks.
Why does Linux say “No space left on device” when df still shows free space?
Then the inodes are very likely exhausted. Check with df -i. In our test, a file system reported “full” at 1 percent usage with 55 MB free, because all 2,000 inodes were used up.
What’s the difference between df and du?
df asks the file system how many blocks are in use. du walks the directories and adds up the sizes of the files it finds. Deleted open files and data hidden under mounts are only counted by df.
How do I find the largest directories on Linux?
sudo du -xh --max-depth=1 / | sort -rh | head shows the largest top-level directories. sudo ncdu -x / is more comfortable and lets you click through directories interactively.
Can I just delete journalctl logs?
Not with rm, but with journalctl --vacuum-size=500M or --vacuum-time=14d. That only removes old, archived journal files and is safe. For a permanent limit, set SystemMaxUse in /etc/systemd/journald.conf.
How do I clean up Docker safely?
Look at docker system df first. Then use docker image prune and docker builder prune for safe cleanup. Avoid docker system prune --volumes, because it deletes the volumes of stopped containers together with their data.
Why can root still write when regular users can’t?
By default ext4 reserves 5 percent of its blocks for root, so system services and admins can still act on a full disk. On data-only disks you can lower the reserve with tune2fs -m 1.
Can a full disk corrupt data? Yes. Programs that empty a file before rewriting it leave an empty file behind when the disk is full. In our test a config file ended up at zero bytes. Databases are usually more robust, but may stop working until space is available again.
How do I prevent the disk from filling up again? Set limits for logs and caches (journald, Docker, binlogs, logrotate), don’t keep backups on the same disk, and set up a warning at 80 to 85 percent, for both bytes and inodes.
Conclusion: a full Linux disk is rarely a mystery if you search in the right order
A full disk on Linux feels chaotic at first, but it almost always follows the same patterns. df -h tells you which file system is affected, df -i whether it’s bytes or inodes, du and ncdu show you the directory, and lsof +L1 explains the cases where the numbers don’t add up.
From our tests we take away three things that hardly any quick guide mentions. First: a deleted file is only freed once nobody holds it open anymore, which is why you empty an active log file with truncate rather than rm. Second: “full” can happen at 1 percent usage when the inodes run out. Third: a full disk isn’t just annoying, it can shrink config files to zero bytes when they’re saved. Keep that in mind, and you’ll clean up first and change things second.
If you’re setting up a new server right now, this is the best moment to add the limits and the warning straight away. How that fits into the rest of the basic setup is shown in our guide Linux server setup.