df14233e629298598736976c5bfcf4a31873745fab# CDDL HEADER START
df14233e629298598736976c5bfcf4a31873745fab# The contents of this file are subject to the terms of the
df14233e629298598736976c5bfcf4a31873745fab# Common Development and Distribution License (the "License").
df14233e629298598736976c5bfcf4a31873745fab# You may not use this file except in compliance with the License.
df14233e629298598736976c5bfcf4a31873745fab# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
df14233e629298598736976c5bfcf4a31873745fab# See the License for the specific language governing permissions
df14233e629298598736976c5bfcf4a31873745fab# and limitations under the License.
df14233e629298598736976c5bfcf4a31873745fab# When distributing Covered Code, include this CDDL HEADER in each
df14233e629298598736976c5bfcf4a31873745fab# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
df14233e629298598736976c5bfcf4a31873745fab# If applicable, add the following below this CDDL HEADER, with the
df14233e629298598736976c5bfcf4a31873745fab# fields enclosed by brackets "[]" replaced with your own identifying
df14233e629298598736976c5bfcf4a31873745fab# information: Portions Copyright [yyyy] [name of copyright owner]
df14233e629298598736976c5bfcf4a31873745fab# CDDL HEADER END
df14233e629298598736976c5bfcf4a31873745fab# Copyright 2008 Sun Microsystems, Inc. All rights reserved.
df14233e629298598736976c5bfcf4a31873745fab# Use is subject to license terms.
df14233e629298598736976c5bfcf4a31873745fabWhy 32-bit libelf is not Large File Aware
df14233e629298598736976c5bfcf4a31873745fab-----------------------------------------
df14233e629298598736976c5bfcf4a31873745fabThe ELF format uses unsigned 32-bit integers for offsets, so the
df14233e629298598736976c5bfcf4a31873745fabtheoretical limit on a 32-bit ELF object is 4GB. However, libelf
df14233e629298598736976c5bfcf4a31873745fabimposes a 2GB limit on the objects it can create. The Solaris
df14233e629298598736976c5bfcf4a31873745fablink-editor and related tools are all based on libelf, so the
df14233e629298598736976c5bfcf4a31873745fab32-bit version of the link-editor also has a 2GB limit, despite
df14233e629298598736976c5bfcf4a31873745fabthe theoretical limit of 4GB.
df14233e629298598736976c5bfcf4a31873745fabLarge file support (LFS) is a half step between the 32 and 64-bit
df14233e629298598736976c5bfcf4a31873745fabworlds, in which an otherwise 32-bit limited process is allowed to
df14233e629298598736976c5bfcf4a31873745fabread and write data to a file that can be larger than 2GB (the extent
df14233e629298598736976c5bfcf4a31873745fabof a signed 32-bit integer, as represented by the system type off_t).
df14233e629298598736976c5bfcf4a31873745fabLFS is useful if the program only needs to access a small subset of
df14233e629298598736976c5bfcf4a31873745fabthe file data at any given time (e.g. /usr/bin/cat). It is less useful
df14233e629298598736976c5bfcf4a31873745fabif the program needs to access a large amount of data at once --- having
df14233e629298598736976c5bfcf4a31873745fabbeen freed from the file limit, the program will simply hit the virtual
df14233e629298598736976c5bfcf4a31873745fabmemory limit (4GB).
df14233e629298598736976c5bfcf4a31873745fabIn particular, the link-editor generally requires twice as much
df14233e629298598736976c5bfcf4a31873745fabmemory as the size of the output object, half to hold the input
df14233e629298598736976c5bfcf4a31873745fabobjects, and half to hold the result. This means that a 32-bit
df14233e629298598736976c5bfcf4a31873745fablink-editor process will hit the 2GB file size limit and the 4GB
df14233e629298598736976c5bfcf4a31873745fabaddress space limit at roughly the same time. As a result, a
df14233e629298598736976c5bfcf4a31873745fablarge file aware 32-bit version of libelf has no significant value.
df14233e629298598736976c5bfcf4a31873745fabDespite this, the question of what it would take to make libelf
df14233e629298598736976c5bfcf4a31873745fablarge file aware comes up from time to time.
df14233e629298598736976c5bfcf4a31873745fabThe first step would be to provide alternative versions of
df14233e629298598736976c5bfcf4a31873745faball public data structures that involve the off_t data type.
df14233e629298598736976c5bfcf4a31873745fabThese structs, found in /usr/include/libelf.h, are:
df14233e629298598736976c5bfcf4a31873745fab * Archive member header
df14233e629298598736976c5bfcf4a31873745fab typedef struct {
df14233e629298598736976c5bfcf4a31873745fab char *ar_name;
df14233e629298598736976c5bfcf4a31873745fab time_t ar_date;
df14233e629298598736976c5bfcf4a31873745fab uid_t ar_uid;
df14233e629298598736976c5bfcf4a31873745fab gid_t ar_gid;
df14233e629298598736976c5bfcf4a31873745fab mode_t ar_mode;
df14233e629298598736976c5bfcf4a31873745fab off_t ar_size;
df14233e629298598736976c5bfcf4a31873745fab char *ar_rawname;
df14233e629298598736976c5bfcf4a31873745fab } Elf_Arhdr;
df14233e629298598736976c5bfcf4a31873745fab * Data descriptor
df14233e629298598736976c5bfcf4a31873745fab typedef struct {
df14233e629298598736976c5bfcf4a31873745fab Elf_Void *d_buf;
df14233e629298598736976c5bfcf4a31873745fab Elf_Type d_type;
df14233e629298598736976c5bfcf4a31873745fab size_t d_size;
df14233e629298598736976c5bfcf4a31873745fab off_t d_off; /* offset into section */
df14233e629298598736976c5bfcf4a31873745fab size_t d_align; /* alignment in section */
df14233e629298598736976c5bfcf4a31873745fab unsigned d_version; /* elf version */
df14233e629298598736976c5bfcf4a31873745fab } Elf_Data;
df14233e629298598736976c5bfcf4a31873745fabAs off_t is a signed type, these alternative versions would have to use
df14233e629298598736976c5bfcf4a31873745faban off64_t type instead.
df14233e629298598736976c5bfcf4a31873745fabIn addition to providing alternative large file aware Elf_Arhdr and
df14233e629298598736976c5bfcf4a31873745fabElf_Data types, it would be necessary to implement large file aware
df14233e629298598736976c5bfcf4a31873745fabversions of the public functions that use them, also found in
df14233e629298598736976c5bfcf4a31873745fab * Function declarations
df14233e629298598736976c5bfcf4a31873745fab unsigned elf_flagdata(Elf_Data *, Elf_Cmd, unsigned);
df14233e629298598736976c5bfcf4a31873745fab Elf_Arhdr *elf_getarhdr(Elf *);
df14233e629298598736976c5bfcf4a31873745fab off_t elf_getbase(Elf *);
df14233e629298598736976c5bfcf4a31873745fab Elf_Data *elf_getdata(Elf_Scn *, Elf_Data *);
df14233e629298598736976c5bfcf4a31873745fab Elf_Data *elf_newdata(Elf_Scn *);
df14233e629298598736976c5bfcf4a31873745fab Elf_Data *elf_rawdata(Elf_Scn *, Elf_Data *);
df14233e629298598736976c5bfcf4a31873745fab off_t elf_update(Elf *, Elf_Cmd);
df14233e629298598736976c5bfcf4a31873745fab Elf_Data *elf32_xlatetof(Elf_Data *, const Elf_Data *, unsigned);
df14233e629298598736976c5bfcf4a31873745fab Elf_Data *elf32_xlatetom(Elf_Data *, const Elf_Data *, unsigned);
df14233e629298598736976c5bfcf4a31873745fab Elf_Data *elf64_xlatetof(Elf_Data *, const Elf_Data *, unsigned);
df14233e629298598736976c5bfcf4a31873745fab Elf_Data *elf64_xlatetom(Elf_Data *, const Elf_Data *, unsigned);
df14233e629298598736976c5bfcf4a31873745fabIt is important to note that these new versions cannot replace the
df14233e629298598736976c5bfcf4a31873745faboriginal definitions. Those must continue to be available to support
bebb829deac32e16136b725d421267d3dceb6cfdRod Evansnon-large-file-aware programs. These new types and functions would be in
df14233e629298598736976c5bfcf4a31873745fabaddition to the pre-existing versions.
df14233e629298598736976c5bfcf4a31873745fabWhen you make code like this large file aware, it is necessary to undertake
df14233e629298598736976c5bfcf4a31873745faba careful analysis of the code to ensure that all the surrounding code uses
df14233e629298598736976c5bfcf4a31873745fabvariable types large enough to handle the increased range. Hence, this work
df14233e629298598736976c5bfcf4a31873745fabis more complicated than simply supplying variants that use a bigger
df14233e629298598736976c5bfcf4a31873745faboff_t and rebuilding --- that is just the first step.
df14233e629298598736976c5bfcf4a31873745fabThere are two standard preprocessor definitions used to control
df14233e629298598736976c5bfcf4a31873745fablarge file support:
df14233e629298598736976c5bfcf4a31873745fab _LARGEFILE64_SOURCE
df14233e629298598736976c5bfcf4a31873745fab _FILE_OFFSET_BITS
df14233e629298598736976c5bfcf4a31873745fabThese preprocessor definitions would be used to determine whether
df14233e629298598736976c5bfcf4a31873745faba given program linked against libelf would see the regular, or
bebb829deac32e16136b725d421267d3dceb6cfdRod Evansthe large file aware versions of the above types and routines.
df14233e629298598736976c5bfcf4a31873745fabThis is the same approach used in other large file capable software,
df14233e629298598736976c5bfcf4a31873745fabsuch as libc.
df14233e629298598736976c5bfcf4a31873745fabFinally, all the applications that rely on libelf would need to be made
df14233e629298598736976c5bfcf4a31873745fablarge file aware. As with libelf itself, there is more to such an effort
df14233e629298598736976c5bfcf4a31873745fabthan recompiling with preprocessor macros set. The code in these
df14233e629298598736976c5bfcf4a31873745fabapplications would need to be examined carefully. Some of these programs
df14233e629298598736976c5bfcf4a31873745fabare very old, and were not originally written with such type portability
df14233e629298598736976c5bfcf4a31873745fabin mind. Such code can be difficult to transition.
df14233e629298598736976c5bfcf4a31873745fabTo work around the 2GB limit in 32-bit libelf:
df14233e629298598736976c5bfcf4a31873745fab - The fundamental limits of a 32-bit address space mean
df14233e629298598736976c5bfcf4a31873745fab that a program this large should be 64-bit. Only a 64-bit
df14233e629298598736976c5bfcf4a31873745fab address space has enough room for that much code, plus the
df14233e629298598736976c5bfcf4a31873745fab stack and heap needed to do useful work with it.
df14233e629298598736976c5bfcf4a31873745fab - The 64-bit version of libelf is also able to process
df14233e629298598736976c5bfcf4a31873745fab 32-bit objects, and does not have a 2GB file size limit.
df14233e629298598736976c5bfcf4a31873745fab Therefore, the 64-bit link-editor can be used to build a 32-bit
df14233e629298598736976c5bfcf4a31873745fab executable which is >2GB. The resulting program will consume over
df14233e629298598736976c5bfcf4a31873745fab half the available address space just to start running. However,
df14233e629298598736976c5bfcf4a31873745fab there may be enough address space left for it to do useful work.
df14233e629298598736976c5bfcf4a31873745fab Note that the 32-bit limit for sharable objects remains at
df14233e629298598736976c5bfcf4a31873745fab 2GB --- imposed by the runtime linker, which is also not large
df14233e629298598736976c5bfcf4a31873745fab file aware.