[DTrace-devel] [PATCH 2/7] types: populate intrinsics from the kernel if possible

[Nick Alcock]

On 15 May 2023, Kris Van Hees verbalised:

> How does this patch fit in the way the type system for D is described in the
> DTrace guide:

Describing the type system for D in this much detail was, frankly, a
mistake (well, it wasn't in the Solaris days, because the Solaris people
controlled the kernel and platform and could make sure the two always
matched, but we don't and never will be able to: can you imagine us
saying to the kernel people that they must change back from using
unsigned char because we have a hardwired list of types in our tracer
that they must conform to? They'd laugh at us, and rightly so).

The type system for D is always, necessarily, the type system of the
underlying kernel: if it's not, you've got a DTrace in which you can't
*use* any types from the kernel, and D is useless (see below). That type
system is not a constant and can change at the whim of kernel developers
and platform ABIs. We should never have hardwired it -- or, rather, we
could certainly hardwire new types, but not types that the kernel
already knows by the same names.

However, by sheer luck the actual description below is still correct for
all kernels I'm aware of other than Itanium (and who cares about
Itanium) -- but that's only because it never specifically defines the
signedness of char (except by implication, where it says "[i]f no sign
qualifier is present, it is assumed that the type is signed", which is
already untrue for char for a number of platforms Linux supports, and as
of 6.3 is untrue for all of them: hence this change). In time this
hardwired set of types can only get more wrong.

So I think this entire documentation section should be rewritten as:
 
> Data Types and Sizes
>
> D provides fundamental data types for integers and floating-point
> constants. Arithmetic may only be performed on integers in D programs.
> Floating-point constants may be used to initialize data structures,
> but floating-point arithmetic is not permitted in D. In Oracle Linux,
> D provides a 64-bit data model for use in writing programs. However, a
> 32-bit data model is not supported. The data model used when executing
> your program is the native data model that is associated with the
> active operating system kernel, which must also be 64-bit.
>
> The names of the integer types and their sizes in the 64-bit data
> model are identical to the type system used by the underlying
> operating system kernel and so can change.  The below are likely types:
>
> Table 2-3 D Integer Data Types
>
> Type Name	64-bit Size
> char            1 byte
> short           2 bytes
> int             4 bytes
> long            8 bytes
> long long       8 bytes
>
> Integer types can be prefixed with the signed or unsigned qualifier.
> If no sign qualifier is present, it is assumed that the type is signed
> (but the signedness of unqualified char is implementation-specific and
> may vary between runs). The D compiler also provides the type aliases
> that are listed in the following table.
>
> Table 2-4 D Integer Type Aliases
>
> Type Name	Description
> int8_t          1-byte signed integer
> int16_t         2-byte signed integer
> int32_t         4-byte signed integer
> int64_t         8-byte signed integer
> intptr_t        Signed integer of size equal to a pointer
> uint8_t         1-byte unsigned integer
> uint16_t        2-byte unsigned integer
> uint32_t        4-byte unsigned integer
> uint64_t        8-byte unsigned integer
> uintptr_t       Unsigned integer of size equal to a pointer
>
> These type aliases are equivalent to using the name of the
> corresponding base type listed in the previous table and are
> appropriately defined for each data model. For example, the uint8_t
> type name is an alias for the type unsigned char. See Type and
> Constant Definitions for information about how to define your own type
> aliases for use in D programs.
>
> Note: The predefined type aliases cannot be used in files that are
> included by the preprocessor.
> 
> D provides floating-point types for compatibility with ANSI C
> declarations and types. Floating-point operators are not supported in
> D, but floating-point data objects can be traced and formatted with
> the printf function. You can use the floating-point types that are
> listed in the following table.
>
> Table 2-5 D Floating-Point Data Types
>
> Type Name	64-bit Size
> float           4 bytes
> double          8 bytes
> long double     16 bytes
>
> D also provides the special type string to represent ASCII strings.
> Strings are discussed in more detail in DTrace Support for Strings.

This is pretty close to what we have already, but we emphasise that the
D types are potentially variable and that the type system in the running
kernel dominates (and we describe the potentially-varying signedness of
char in particular: not allowing for that was *definitely* a mistake,
since the C Standard and even then-existing Unix implementations never
required a specific signedness for char, but of course Solaris used the
one they nailed down in there).

> Most specifically, DTrace seems to be defining its own type system (obviously
> based on standard systems) rather than using whatever the runtime system is
> as presented by the runtime kernel.  As such, it seems to me that this patch
> is not OK because it makes the type system dependent on the kernel rather than
> D (DTrace) having its own well defined type system.

But that's already the case! DTrace already works by copying all types
you reference from vmlinux` or any other dict into the D dict as you use
them -- so if they *don't* use identical underlying type definitions,
disaster results (at the very least you get told ludicrous things like
that psinfo.comm is unusable because char[16] != char[16], but I've also
seen outright crashes). So the D dict is necessarily a derivative of the
kernel's: types they have in common must agree, and the code silently
assumes this. It's just that we don't actually enforce this, we just
*hope*. After this change, we can guarantee that the two type systems
always use identical types where necessary.

The only reason we haven't seen disaster already is because the kernel's
type system *happens* not to change very often and *happens* to be very
similar to our hardwired list -- but it is *not* set in stone, it *can*
change, it is its own freestanding ABI, and we must follow it. It's
either that, or never use any kernel types in D, and to me that seems
likely to render DTrace immediately useless. (You can't start DTrace on
6.3+ without these patches or something cruder to adapt our hardwired
type list to follow the kernel's -- and if we're doing the latter, why
not make our lives easier and actually *copy in* the kernel's base
types, just like we already do with all the other kernel types we use?
Then we never need to worry about this ever again.)

The kernel types and the D types obviously have to end up in the same
set of CTF dicts, or you can't use them in the same program, so the
"aggressive copy in in the D dict" approach D uses for all types that
aren't in the hardwired list is obviously essential -- and if we don't
ultimately end up deriving the base types those types use from the
kernel's type system, ctf_add_type() will end up producing an
inconsistent type system, either with two types named (say) "char" with
different definitions, or with the alleged kernel type being derived
from a D type with the wrong definition, leading to obvious problems
trying to actually *use* them. It so happens you get the first right now
for char -- for typedefs, you'd get the other problematic alternative (I
think we already have some subtly wrong typedefs in the hardwired list:
this patch fixes them too.)

(One alternative is to not have a C dict at all and just hang the D dict
directly off the shared_ctf dict, but that's much more tricky in all
sorts of ways and I'd rather not go there. Copying things into the D
dict as we use them keeps us sealed off to a degree from the huge
churning universe of vmlinux.ctfa and also means we don't always need to
load everything in the CTFA, both of which are useful -- but that
doesn't mean we are *independent* of it. It contributes types to the D
dict, so our base types must be the same.)