希望深入了解 Linux 上动态内存分配的内部结构,我能找到的最好的文章是一篇题为理解 glibc malloc的文章。解释虽然详细,但不太好理解(对我来说)。特别是,由于零散的写作风格,我无法理解所涉及的所有数据结构,它以分段方式提供信息。任何人都可以提供一份或多份详细介绍同一主题的参考资料吗?
glibc malloc 是如何工作的?
逆向工程
linux
记忆
库
2021-06-23 04:40:48
2个回答
对于如何理解动态内存分配(malloc的,免费的,释放calloc,realloc的库函数),真正起作用的是没有替代阅读源代码的malloc()
。评论很好:
对块的评论:
/*
1056 malloc_chunk details:
1057
1058 (The following includes lightly edited explanations by Colin Plumb.)
1059
1060 Chunks of memory are maintained using a `boundary tag' method as
1061 described in e.g., Knuth or Standish. (See the paper by Paul
1062 Wilson ftp://ftp.cs.utexas.edu/pub/garbage/allocsrv.ps for a
1063 survey of such techniques.) Sizes of free chunks are stored both
1064 in the front of each chunk and at the end. This makes
1065 consolidating fragmented chunks into bigger chunks very fast. The
1066 size fields also hold bits representing whether chunks are free or
1067 in use.
1068
1069 An allocated chunk looks like this:
1070
1071
1072 chunk-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1073 | Size of previous chunk, if unallocated (P clear) |
1074 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1075 | Size of chunk, in bytes |A|M|P|
1076 mem-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1077 | User data starts here... .
1078 . .
1079 . (malloc_usable_size() bytes) .
1080 . |
1081 nextchunk-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1082 | (size of chunk, but used for application data) |
1083 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1084 | Size of next chunk, in bytes |A|0|1|
1085 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1086
1087 Where "chunk" is the front of the chunk for the purpose of most of
1088 the malloc code, but "mem" is the pointer that is returned to the
1089 user. "Nextchunk" is the beginning of the next contiguous chunk.
1090
1091 Chunks always begin on even word boundaries, so the mem portion
1092 (which is returned to the user) is also on an even word boundary, and
1093 thus at least double-word aligned.
1094
1095 Free chunks are stored in circular doubly-linked lists, and look like this:
1096
1097 chunk-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1098 | Size of previous chunk, if unallocated (P clear) |
1099 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1100 `head:' | Size of chunk, in bytes |A|0|P|
1101 mem-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1102 | Forward pointer to next chunk in list |
1103 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1104 | Back pointer to previous chunk in list |
1105 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1106 | Unused space (may be 0 bytes long) .
1107 . .
1108 . |
1109 nextchunk-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1110 `foot:' | Size of chunk, in bytes |
1111 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1112 | Size of next chunk, in bytes |A|0|0|
1113 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1114
1115 The P (PREV_INUSE) bit, stored in the unused low-order bit of the
1116 chunk size (which is always a multiple of two words), is an in-use
1117 bit for the *previous* chunk. If that bit is *clear*, then the
1118 word before the current chunk size contains the previous chunk
1119 size, and can be used to find the front of the previous chunk.
1120 The very first chunk allocated always has this bit set,
1121 preventing access to non-existent (or non-owned) memory. If
1122 prev_inuse is set for any given chunk, then you CANNOT determine
1123 the size of the previous chunk, and might even get a memory
1124 addressing fault when trying to do so.
1125
1126 The A (NON_MAIN_ARENA) bit is cleared for chunks on the initial,
1127 main arena, described by the main_arena variable. When additional
1128 threads are spawned, each thread receives its own arena (up to a
1129 configurable limit, after which arenas are reused for multiple
1130 threads), and the chunks in these arenas have the A bit set. To
1131 find the arena for a chunk on such a non-main arena, heap_for_ptr
1132 performs a bit mask operation and indirection through the ar_ptr
1133 member of the per-heap header heap_info (see arena.c).
1134
1135 Note that the `foot' of the current chunk is actually represented
1136 as the prev_size of the NEXT chunk. This makes it easier to
1137 deal with alignments etc but can be very confusing when trying
1138 to extend or adapt this code.
1139
1140 The three exceptions to all this are:
1141
1142 1. The special chunk `top' doesn't bother using the
1143 trailing size field since there is no next contiguous chunk
1144 that would have to index off it. After initialization, `top'
1145 is forced to always exist. If it would become less than
1146 MINSIZE bytes long, it is replenished.
1147
1148 2. Chunks allocated via mmap, which have the second-lowest-order
1149 bit M (IS_MMAPPED) set in their size fields. Because they are
1150 allocated one-by-one, each must contain its own trailing size
1151 field. If the M bit is set, the other bits are ignored
1152 (because mmapped chunks are neither in an arena, nor adjacent
1153 to a freed chunk). The M bit is also used for chunks which
1154 originally came from a dumped heap via malloc_set_state in
1155 hooks.c.
1156
1157 3. Chunks in fastbins are treated as allocated chunks from the
1158 point of view of the chunk allocator. They are consolidated
1159 with their neighbors only in bulk, in malloc_consolidate.
1160 */
关于内部数据结构的评论:
/*
1313 -------------------- Internal data structures --------------------
1314
1315 All internal state is held in an instance of malloc_state defined
1316 below. There are no other static variables, except in two optional
1317 cases:
1318 * If USE_MALLOC_LOCK is defined, the mALLOC_MUTEx declared above.
1319 * If mmap doesn't support MAP_ANONYMOUS, a dummy file descriptor
1320 for mmap.
1321
1322 Beware of lots of tricks that minimize the total bookkeeping space
1323 requirements. The result is a little over 1K bytes (for 4byte
1324 pointers and size_t.)
1325 */
1326
1327 /*
1328 Bins
1329
1330 An array of bin headers for free chunks. Each bin is doubly
1331 linked. The bins are approximately proportionally (log) spaced.
1332 There are a lot of these bins (128). This may look excessive, but
1333 works very well in practice. Most bins hold sizes that are
1334 unusual as malloc request sizes, but are more usual for fragments
1335 and consolidated sets of chunks, which is what these bins hold, so
1336 they can be found quickly. All procedures maintain the invariant
1337 that no consolidated chunk physically borders another one, so each
1338 chunk in a list is known to be preceeded and followed by either
1339 inuse chunks or the ends of memory.
1340
1341 Chunks in bins are kept in size order, with ties going to the
1342 approximately least recently used chunk. Ordering isn't needed
1343 for the small bins, which all contain the same-sized chunks, but
1344 facilitates best-fit allocation for larger chunks. These lists
1345 are just sequential. Keeping them in order almost never requires
1346 enough traversal to warrant using fancier ordered data
1347 structures.
1348
1349 Chunks of the same size are linked with the most
1350 recently freed at the front, and allocations are taken from the
1351 back. This results in LRU (FIFO) allocation order, which tends
1352 to give each chunk an equal opportunity to be consolidated with
1353 adjacent freed chunks, resulting in larger free chunks and less
1354 fragmentation.
1355
1356 To simplify use in double-linked lists, each bin header acts
1357 as a malloc_chunk. This avoids special-casing for headers.
1358 But to conserve space and improve locality, we allocate
1359 only the fd/bk pointers of bins, and then use repositioning tricks
1360 to treat these as the fields of a malloc_chunk*.
1361 */
的作者之一malloc()
,Doug Lea,写了一篇名为“ A Memory Allocator ”的文章,描述了 malloc 是如何工作的(注意这篇文章是 2000 年的,所以会有一些过时的信息)。
从文章:
块:
垃圾箱:
另一个资源是Michael Kerrisk的“Linux 编程接口”的第 7 章:“内存分配” 。TLPI 是我所遇到过的任何类型的最佳参考,并且推荐得不够高。
这里的实施方式的图示malloc()
,并free()
从TLPI:
最后一点,malloc()
是对brk()
和sbrk()
系统调用的包装,它通过更改程序中断的位置来调整堆的大小。
来自来源中的评论:
901 In the new situation, brk() and mmap space is shared and there are no
902 artificial limits on brk size imposed by the kernel. What is more,
903 applications have started using transient allocations larger than the
904 128Kb as was imagined in 2001.
905
906 The price for mmap is also high now; each time glibc mmaps from the
907 kernel, the kernel is forced to zero out the memory it gives to the
908 application. Zeroing memory is expensive and eats a lot of cache and
909 memory bandwidth. This has nothing to do with the efficiency of the
910 virtual memory system, by doing mmap the kernel just has no choice but
911 to zero.
912
913 In 2001, the kernel had a maximum size for brk() which was about 800
914 megabytes on 32 bit x86, at that point brk() would hit the first
915 mmaped shared libaries and couldn't expand anymore. With current 2.6
916 kernels, the VA space layout is different and brk() and mmap
917 both can span the entire heap at will.
918
919 Rather than using a static threshold for the brk/mmap tradeoff,
920 we are now using a simple dynamic one. The goal is still to avoid
921 fragmentation. The old goals we kept are
922 1) try to get the long lived large allocations to use mmap()
923 2) really large allocations should always use mmap()
924 and we're adding now:
925 3) transient allocations should use brk() to avoid forcing the kernel
926 having to zero memory over and over again
927
928 The implementation works with a sliding threshold, which is by default
929 limited to go between 128Kb and 32Mb (64Mb for 64 bitmachines) and starts
930 out at 128Kb as per the 2001 default.
931
932 This allows us to satisfy requirement 1) under the assumption that long
933 lived allocations are made early in the process' lifespan, before it has
934 started doing dynamic allocations of the same size (which will
935 increase the threshold).
936
937 The upperbound on the threshold satisfies requirement 2)
938
939 The threshold goes up in value when the application frees memory that was
940 allocated with the mmap allocator. The idea is that once the application
941 starts freeing memory of a certain size, it's highly probable that this is
942 a size the application uses for transient allocations. This estimator
943 is there to satisfy the new third requirement.
944
945 */
关于在软件安全中利用堆,有几个很好的参考资料,我最喜欢的一个可能是来自LiveOverflow的“二进制黑客课程” 。
您可以查看以下讲座,了解堆管理的简化方法(使用来自Exploit-Exercises的Protostar练习集):
- 0x14 - 堆:做
malloc()
什么? - 0x15 - 堆:如何利用堆溢出
- 0x16 - 堆:释放后使用漏洞是如何工作的?
- 0x17 - 堆:从前
free()
- 0x18 - 堆:dlmalloc
unlink()
漏洞利用
并且:
- 0x1F - [Live] 远程老式 dlmalloc 堆漏洞利用
但是,来自sploitfun的博客文章是我在网上看到的关于这个特定主题的最准确的文章之一。一旦你对基本原理有了足够的了解,我会建议你回到 sploitfun 的文章。
其它你可能感兴趣的问题