| Type | Default | Details | |
|---|---|---|---|
| a | description of param a | ||
| b | bool | True | description of param b |
| c | str | None | |
| Returns | int |
| Type | Default | Details | |
|---|---|---|---|
| a | |||
| b | int | param b | |
| c | str | foo | param c |
| Returns | str | Result of doing it |
| Type | Default | Details | |
|---|---|---|---|
| a | description of param a | ||
| b | bool | True | description of param b |
| c | str | None |
| Type | Default | Details | |
|---|---|---|---|
| c | int | description of param c | |
| d | bool | True | description of param d |
However, when you inherhit from a class that defines `__new__` or a
metaclass that defines `__call__` this obfuscates the signature by
overriding your class with the signature of `__new__`, which prevents
tab completion from displaying useful information:
``` python
class Foo:
def __new__(self, **args): pass
class Bar(Foo):
def __init__(self, d, e, f): pass
inspect.signature(Bar)
```
Finally, the signature and tab completion can be restored by inheriting
from the metaclass
[`FixSigMeta`](https://fastcore.fast.ai/meta.html#fixsigmeta) as shown
below:
``` python
class Bar(Foo, metaclass=FixSigMeta):
def __init__(self, d, e, f): pass
test_sig(Bar, '(d, e, f)')
inspect.signature(Bar)
```
If you need to define a metaclass that overrides `__call__` (as done in
[`PrePostInitMeta`](https://fastcore.fast.ai/meta.html#prepostinitmeta)),
you need to inherit from
[`FixSigMeta`](https://fastcore.fast.ai/meta.html#fixsigmeta) instead of
`type` when constructing the metaclass to preserve the signature in
`__init__`. Be careful not to override `__new__` when doing this:
``` python
class TestMeta(FixSigMeta):
# __new__ comes from FixSigMeta
def __call__(cls, *args, **kwargs): pass
class T(metaclass=TestMeta):
def __init__(self, a, b): pass
test_sig(T, '(a, b)')
```
On the other hand, if you fail to inherit from
[`FixSigMeta`](https://fastcore.fast.ai/meta.html#fixsigmeta) when
inheriting from a metaclass that overrides `__call__`, your signature
will reflect that of `__call__` instead (which is often undesirable):
``` python
class GenericMeta(type):
"A boilerplate metaclass that doesn't do anything for testing."
def __new__(cls, name, bases, dict):
return super().__new__(cls, name, bases, dict)
def __call__(cls, *args, **kwargs): pass
class T2(metaclass=GenericMeta):
def __init__(self, a, b): pass
# We can avoid this by inheriting from the metaclass `FixSigMeta`
test_sig(T2, '(*args, **kwargs)')
```
------------------------------------------------------------------------
source
### PrePostInitMeta
``` python
def PrePostInitMeta(
args:VAR_POSITIONAL, kwargs:VAR_KEYWORD
):
```
*A metaclass that calls optional `__pre_init__` and `__post_init__`
methods*
`__pre_init__` and `__post_init__` are useful for initializing variables
or performing tasks prior to or after `__init__` being called,
respectively. Fore example:
``` python
class _T(metaclass=PrePostInitMeta):
def __pre_init__(self): self.a = 0;
def __init__(self,b=0): self.b = self.a + 1; assert self.b==1
def __post_init__(self): self.c = self.b + 2; assert self.c==3
t = _T()
test_eq(t.a, 0) # set with __pre_init__
test_eq(t.b, 1) # set with __init__
test_eq(t.c, 3) # set with __post_init__
```
One use for
[`PrePostInitMeta`](https://fastcore.fast.ai/meta.html#prepostinitmeta)
is avoiding the `__super__().__init__()` boilerplate associated with
subclassing, such as used in
[`AutoInit`](https://fastcore.fast.ai/meta.html#autoinit).
------------------------------------------------------------------------
source
### AutoInit
``` python
def AutoInit(
args:VAR_POSITIONAL, kwargs:VAR_KEYWORD
):
```
*Same as `object`, but no need for subclasses to call
`super().__init__`*
This is normally used as a
[mixin](https://www.residentmar.io/2019/07/07/python-mixins.html), eg:
``` python
class TestParent():
def __init__(self): self.h = 10
class TestChild(AutoInit, TestParent):
def __init__(self): self.k = self.h + 2
t = TestChild()
test_eq(t.h, 10) # h=10 is initialized in the parent class
test_eq(t.k, 12)
```
------------------------------------------------------------------------
source
### NewChkMeta
``` python
def NewChkMeta(
args:VAR_POSITIONAL, kwargs:VAR_KEYWORD
):
```
*Metaclass to avoid recreating object passed to constructor*
[`NewChkMeta`](https://fastcore.fast.ai/meta.html#newchkmeta) is used
when an object of the same type is the first argument to your class’s
constructor (i.e. the `__init__` function), and you would rather it not
create a new object but point to the same exact object.
This is used in [`L`](https://fastcore.fast.ai/foundation.html#l), for
example, to avoid creating a new object when the object is already of
type [`L`](https://fastcore.fast.ai/foundation.html#l). This allows the
users to defenisvely instantiate an
[`L`](https://fastcore.fast.ai/foundation.html#l) object and just return
a reference to the same object if it already happens to be of type
[`L`](https://fastcore.fast.ai/foundation.html#l).
For example, the below class `_T` **optionally** accepts an object `o`
as its first argument. A new object is returned upon instantiation per
usual:
``` python
class _T():
"Testing"
def __init__(self, o):
# if `o` is not an object without an attribute `foo`, set foo = 1
self.foo = getattr(o,'foo',1)
```
``` python
t = _T(3)
test_eq(t.foo,1) # 1 was not of type _T, so foo = 1
t2 = _T(t) #t1 is of type _T
assert t is not t2 # t1 and t2 are different objects
```
However, if we want `_T` to return a reference to the same object when
passed an an object of type `_T` we can inherit from the
[`NewChkMeta`](https://fastcore.fast.ai/meta.html#newchkmeta) class as
illustrated below:
``` python
class _T(metaclass=NewChkMeta):
"Testing with metaclass NewChkMeta"
def __init__(self, o=None, b=1):
# if `o` is not an object without an attribute `foo`, set foo = 1
self.foo = getattr(o,'foo',1)
self.b = b
```
We can now test `t` and `t2` are now pointing at the same object when
using this new definition of `_T`:
``` python
t = _T(3)
test_eq(t.foo,1) # 1 was not of type _T, so foo = 1
t2 = _T(t) # t2 will now reference t
test_is(t, t2) # t and t2 are the same object
t2.foo = 5 # this will also change t.foo to 5 because it is the same object
test_eq(t.foo, 5)
test_eq(t2.foo, 5)
```
However, there is one exception to how
[`NewChkMeta`](https://fastcore.fast.ai/meta.html#newchkmeta) works.
**If you pass any additional arguments in the constructor a new object
is returned**, even if the first object is of the same type. For
example, consider the below example where we pass the additional
argument `b` into the constructor:
``` python
t3 = _T(t, b=1)
assert t3 is not t
t4 = _T(t) # without any arguments the constructor will return a reference to the same object
assert t4 is t
```
Finally, it should be noted that
[`NewChkMeta`](https://fastcore.fast.ai/meta.html#newchkmeta) as well as
all other metaclases in this section, inherit from
[`FixSigMeta`](https://fastcore.fast.ai/meta.html#fixsigmeta). This
means class signatures will always be preserved when inheriting from
this metaclass (see docs for
[`FixSigMeta`](https://fastcore.fast.ai/meta.html#fixsigmeta) for more
details):
``` python
test_sig(_T, '(o=None, b=1)')
```
------------------------------------------------------------------------
source
### BypassNewMeta
``` python
def BypassNewMeta(
args:VAR_POSITIONAL, kwargs:VAR_KEYWORD
):
```
*Metaclass: casts `x` to this class if it’s of type `cls._bypass_type`*
[`BypassNewMeta`](https://fastcore.fast.ai/meta.html#bypassnewmeta) is
identical to
[`NewChkMeta`](https://fastcore.fast.ai/meta.html#newchkmeta), except
for checking for a class as the same type, we instead check for a class
of type specified in attribute `_bypass_type`.
In NewChkMeta, objects of the same type passed to the constructor
(without arguments) would result into a new variable referencing the
same object. However, with
[`BypassNewMeta`](https://fastcore.fast.ai/meta.html#bypassnewmeta) this
only occurs if the type matches the `_bypass_type` of the class you are
defining:
``` python
class _TestA: pass
class _TestB: pass
class _T(_TestA, metaclass=BypassNewMeta):
_bypass_type=_TestB
def __init__(self,x): self.x=x
```
In the below example, `t` does not refer to `t2` because `t` is of type
`_TestA` while `_T._bypass_type` is of type `TestB`:
``` python
t = _TestA()
t2 = _T(t)
assert t is not t2
```
However, if `t` is set to `_TestB` to match `_T._bypass_type`, then both
`t` and `t2` will refer to the same object.
``` python
t = _TestB()
t2 = _T(t)
t2.new_attr = 15
test_is(t, t2)
# since t2 just references t these will be the same
test_eq(t.new_attr, t2.new_attr)
# likewise, chaning an attribute on t will also affect t2 because they both point to the same object.
t.new_attr = 9
test_eq(t2.new_attr, 9)
```
## Metaprogramming
------------------------------------------------------------------------
source
### empty2none
``` python
def empty2none(
p
):
```
*Replace `Parameter.empty` with `None`*
------------------------------------------------------------------------
source
### anno_dict
``` python
def anno_dict(
f
):
```
*`__annotation__ dictionary with`empty`cast to`None\`, returning empty
if doesn’t exist*
``` python
def _f(a:int, b:L)->str: ...
test_eq(anno_dict(_f), {'a': int, 'b': L, 'return': str})
```
------------------------------------------------------------------------
source
### use_kwargs_dict
``` python
def use_kwargs_dict(
keep:bool=False, kwargs:VAR_KEYWORD
):
```
*Decorator: replace `**kwargs` in signature with `names` params*
Replace all `**kwargs` with named arguments like so:
``` python
@use_kwargs_dict(y=1,z=None)
def foo(a, b=1, **kwargs): pass
test_sig(foo, '(a, b=1, *, y=1, z=None)')
```
Add named arguments, but optionally keep `**kwargs` by setting
`keep=True`:
``` python
@use_kwargs_dict(y=1,z=None, keep=True)
def foo(a, b=1, **kwargs): pass
test_sig(foo, '(a, b=1, *, y=1, z=None, **kwargs)')
```
------------------------------------------------------------------------
source
### use_kwargs
``` python
def use_kwargs(
names, keep:bool=False
):
```
*Decorator: replace `**kwargs` in signature with `names` params*
[`use_kwargs`](https://fastcore.fast.ai/meta.html#use_kwargs) is
different than
[`use_kwargs_dict`](https://fastcore.fast.ai/meta.html#use_kwargs_dict)
as it only replaces `**kwargs` with named parameters without any default
values:
``` python
@use_kwargs(['y', 'z'])
def foo(a, b=1, **kwargs): pass
test_sig(foo, '(a, b=1, *, y=None, z=None)')
```
You may optionally keep the `**kwargs` argument in your signature by
setting `keep=True`:
``` python
@use_kwargs(['y', 'z'], keep=True)
def foo(a, *args, b=1, **kwargs): pass
test_sig(foo, '(a, *args, b=1, y=None, z=None, **kwargs)')
```
``` python
from fastcore.basics import *
```
``` python
listify('a')
```
['a']
------------------------------------------------------------------------
source
### delegates
``` python
def delegates(
to:function=None, # Delegatee
keep:bool=False, # Keep `kwargs` in decorated function?
but:list=None, # Exclude these parameters from signature
sort_args:bool=False, # Sort arguments alphabetically, doesn't work with call_parse
):
```
*Decorator: replace `**kwargs` in signature with params from `to`*
``` python
def baz(a, b:int=2, c:int=3): return a + b + c
def foo(c, a, **kwargs):
return c + baz(a, **kwargs)
assert foo(c=1, a=1) == 7
```
The problem with this approach is the api for `foo` is obfuscated. Users
cannot introspect what the valid arguments for `**kwargs` are without
reading the source code. When a user tries tries to introspect the
signature of `foo`, they are presented with this:
``` python
inspect.signature(foo)
```
> **Boolean Arguments Default to False**
>
> Arguments of type `bool` or
> [`store_true`](https://fastcore.fast.ai/script.html#store_true)
> default to `False` regardless of whether you provide a default or not.
> Use [`bool_arg`](https://fastcore.fast.ai/script.html#bool_arg) as the
> type instead of `bool` if you want to set a default value of True. For
> example:
>
> ``` python
> @call_parse
> def main(msg:str="Hi", # The message
> upper:bool_arg=True): # Convert to uppercase?
> ```
## Param annotations
If you want to use the full power of `argparse`, you can do so by using
[`Param`](https://fastcore.fast.ai/script.html#param) annotations
instead of type annotations and
[docments](https://fastcore.fast.ai/docments.html), like so:
``` python
from fastcore.script import *
@call_parse
def main(msg:Param("The message", str),
upper:Param("Convert to uppercase?", store_true)):
"Print `msg`, optionally converting to uppercase"
print(msg.upper() if upper else msg)
```
If you use this approach, then each parameter in your function should
have an annotation `Param(...)` (as in the example above). You can pass
the following when calling
[`Param`](https://fastcore.fast.ai/script.html#param):
`help`,`type`,`opt`,`action`,`nargs`,`const`,`choices`,`required` .
Except for `opt`, all of these are just passed directly to `argparse`,
so you have all the power of that module at your disposal. Generally
you’ll want to pass at least `help` (since this is provided as the help
string for that parameter) and `type` (to ensure that you get the type
of data you expect). `opt` is a bool that defines whether a param is
optional or required (positional) - but you’ll generally not need to set
this manually, because fastcore.script will set it for you automatically
based on *default* values.
## setuptools scripts
There’s a really nice feature of pip/setuptools that lets you create
commandline scripts directly from functions, makes them available in the
`PATH`, and even makes your scripts cross-platform (e.g. in Windows it
creates an exe). fastcore.script supports this feature too. The trick to
making a function available as a script is to add a `console_scripts`
section to your setup file, of the form:
`script_name=module:function_name`. E.g. in this case we use:
`test_fastcore.script=fastcore.script.test_cli:main`. With this, you can
then just type `test_fastcore.script` at any time, from any directory,
and your script will be called (once it’s installed using one of the
methods below).
You don’t actually have to write a `setup.py` yourself. Instead, just
use [nbdev](https://nbdev.fast.ai). Then modify `settings.ini` as
appropriate for your module/script. To install your script directly, you
can type `pip install -e .`. Your script, when installed this way (it’s
called an [editable
install](http://codumentary.blogspot.com/2014/11/python-tip-of-year-pip-install-editable.html)),
will automatically be up to date even if you edit it - there’s no need
to reinstall it after editing. With nbdev you can even make your module
and script available for installation directly from pip and conda by
running `make release`.
## API details
------------------------------------------------------------------------
source
### store_true
``` python
def store_true(
):
```
*Placeholder to pass to
[`Param`](https://fastcore.fast.ai/script.html#param) for
[`store_true`](https://fastcore.fast.ai/script.html#store_true) action*
------------------------------------------------------------------------
source
### store_false
``` python
def store_false(
):
```
*Placeholder to pass to
[`Param`](https://fastcore.fast.ai/script.html#param) for
[`store_false`](https://fastcore.fast.ai/script.html#store_false)
action*
------------------------------------------------------------------------
source
### bool_arg
``` python
def bool_arg(
v
):
```
*Use as `type` for [`Param`](https://fastcore.fast.ai/script.html#param)
to get `bool` behavior*
------------------------------------------------------------------------
source
### clean_type_str
``` python
def clean_type_str(
x:str
):
```
``` python
class Test: pass
test_eq(clean_type_str(argparse.ArgumentParser), 'argparse.ArgumentParser')
test_eq(clean_type_str(Test), 'Test')
test_eq(clean_type_str(int), 'int')
test_eq(clean_type_str(float), 'float')
test_eq(clean_type_str(store_false), 'store_false')
```
------------------------------------------------------------------------
source
### Param
``` python
def Param(
help:str='', type:NoneType=None, opt:bool=True, action:NoneType=None, nargs:NoneType=None, const:NoneType=None,
choices:NoneType=None, required:NoneType=None, default:NoneType=None, version:NoneType=None
):
```
*A parameter in a function used in
[`anno_parser`](https://fastcore.fast.ai/script.html#anno_parser) or
[`call_parse`](https://fastcore.fast.ai/script.html#call_parse)*
``` python
test_eq(repr(Param("Help goes here")), '
The documentation also contains links to any related functions or
classes, which appear like this:
[`coll_repr`](https://fastcore.fast.ai/foundation.html#coll_repr) (in
the notebook itself you will just see a word with back-ticks around it;
the links are auto-generated in the documentation site). The
documentation will generally show one or more examples of use, along
with any background context necessary to understand them. As you’ll see,
the examples for each function and method are shown as tests, rather
than example outputs, so let’s start by explaining that.
### Testing
fastcore’s testing module is designed to work well with
[nbdev](https://nbdev.fast.ai), which is a full literate programming
environment built on Jupyter Notebooks. That means that your tests,
docs, and code all live together in the same notebook. fastcore and
nbdev’s approach to testing starts with the premise that all your tests
should pass. If one fails, no more tests in a notebook are run.
Tests look like this:
``` python
test_eq(coll_repr(range(1000), 5), '(#1000) [0, 1, 2, 3, 4...]')
```
That’s an example from the docs for
[`coll_repr`](https://fastcore.fast.ai/foundation.html#coll_repr). As
you see, it’s not showing you the output directly. Here’s what that
would look like:
``` python
coll_repr(range(1000), 5)
```
'(#1000) [0, 1, 2, 3, 4...]'
So, the test is actually showing you what the output looks like, because
if the function call didn’t return `'(#1000) [0,1,2,3,4...]'`, then the
test would have failed.
So every test shown in the docs is also showing you the behavior of the
library — and vice versa!
Test functions always start with `test_`, and then follow with the
operation being tested. So
[`test_eq`](https://fastcore.fast.ai/test.html#test_eq) tests for
equality (as you saw in the example above). This includes tests for
equality of arrays and tensors, lists and generators, and many more:
``` python
test_eq([0,1,2,3], np.arange(4))
```
When a test fails, it prints out information about what was expected:
``` python
test_eq([0,1,2,3], np.arange(3))
```
----
AssertionError: ==:
[0, 1, 2, 3]
[0 1 2]
If you want to check that objects are the same type, rather than the
just contain the same collection, use
[`test_eq_type`](https://fastcore.fast.ai/test.html#test_eq_type).
You can test with any comparison function using
[`test`](https://fastcore.fast.ai/test.html#test), e.g test whether an
object is less than:
``` python
test(2, 3, operator.lt)
```
You can even test that exceptions are raised:
``` python
def divide_zero(): return 1/0
test_fail(divide_zero)
```
…and test that things are printed to stdout:
``` python
test_stdout(lambda: print('hi'), 'hi')
```
### Foundations
fast.ai is unusual in that we often use
[mixins](https://en.wikipedia.org/wiki/Mixin) in our code. Mixins are
widely used in many programming languages, such as Ruby, but not so much
in Python. We use mixins to attach new behavior to existing libraries,
or to allow modules to add new behavior to our own classes, such as in
extension modules. One useful example of a mixin we define is
[`Path.ls`](https://fastcore.fast.ai/xtras.html#path.ls), which lists a
directory and returns an
[`L`](https://fastcore.fast.ai/foundation.html#l) (an extended list
class which we’ll discuss shortly):
``` python
p = Path('images')
p.ls()
```
[Path('images/mnist3.png'), Path('images/att_00000.png'), Path('images/att_00005.png'), Path('images/att_00007.png'), Path('images/att_00006.png'), Path('images/puppy.jpg')]
You can easily add you own mixins with the
[`patch`](https://fastcore.fast.ai/basics.html#patch)
[decorator](https://realpython.com/primer-on-python-decorators/), which
takes advantage of Python 3 [function
annotations](https://www.python.org/dev/peps/pep-3107/#parameters) to
say what class to patch:
``` python
@patch
def num_items(self:Path): return len(self.ls())
p.num_items()
```
6
We also use `**kwargs` frequently. In python `**kwargs` in a parameter
like means “*put any additional keyword arguments into a dict called
`kwargs`*”. Normally, using `kwargs` makes an API quite difficult to
work with, because it breaks things like tab-completion and popup lists
of signatures. `utils` provides
[`use_kwargs`](https://fastcore.fast.ai/meta.html#use_kwargs) and
[`delegates`](https://fastcore.fast.ai/meta.html#delegates) to avoid
this problem. See our [detailed article on
delegation](https://www.fast.ai/2019/08/06/delegation/) on this topic.
[`GetAttr`](https://fastcore.fast.ai/basics.html#getattr) solves a
similar problem (and is also discussed in the article linked above):
it’s allows you to use Python’s exceptionally useful
[`__getattr__`](https://fastcore.fast.ai/xml.html#__getattr__) magic
method, but avoids the problem that normally in Python tab-completion
and docs break when using this. For instance, you can see here that
Python’s `dir` function, which is used to find the attributes of a
python object, finds everything inside the `self.default` attribute
here:
``` python
class Author:
def __init__(self, name): self.name = name
class ProductPage(GetAttr):
_default = 'author'
def __init__(self,author,price,cost): self.author,self.price,self.cost = author,price,cost
p = ProductPage(Author("Jeremy"), 1.50, 0.50)
[o for o in dir(p) if not o.startswith('_')]
```
['author', 'cost', 'name', 'price']
Looking at that `ProductPage` example, it’s rather verbose and
duplicates a lot of attribute names, which can lead to bugs later if you
change them only in one place. `fastcore` provides
[`store_attr`](https://fastcore.fast.ai/basics.html#store_attr) to
simplify this common pattern. It also provides
[`basic_repr`](https://fastcore.fast.ai/basics.html#basic_repr) to give
simple objects a useful `repr`:
``` python
class ProductPage:
def __init__(self,author,price,cost): store_attr()
__repr__ = basic_repr('author,price,cost')
ProductPage("Jeremy", 1.50, 0.50)
```
ProductPage(author='Jeremy', price=1.5, cost=0.5)
One of the most interesting `fastcore` functions is the
[`funcs_kwargs`](https://fastcore.fast.ai/meta.html#funcs_kwargs)
decorator. This allows class behavior to be modified without
sub-classing. This can allow folks that aren’t familiar with
object-oriented programming to customize your class more easily. Here’s
an example of a class that uses
[`funcs_kwargs`](https://fastcore.fast.ai/meta.html#funcs_kwargs):
``` python
@funcs_kwargs
class T:
_methods=['some_method']
def __init__(self, **kwargs): assert not kwargs, f'Passed unknown args: {kwargs}'
p = T(some_method = print)
p.some_method("hello")
```
hello
The `assert not kwargs` above is used to ensure that the user doesn’t
pass an unknown parameter (i.e one that’s not in `_methods`). `fastai`
uses [`funcs_kwargs`](https://fastcore.fast.ai/meta.html#funcs_kwargs)
in many places, for instance, you can customize any part of a
`DataLoader` by passing your own methods.
`fastcore` also provides many utility functions that make a Python
programmer’s life easier, in `fastcore.utils`. We won’t look at many
here, since you can easily look at the docs yourself. To get you
started, have a look at the docs for
[`chunked`](https://fastcore.fast.ai/basics.html#chunked) (remember, if
you’re in a notebook, type `doc(chunked)`), which is a handy function
for creating lazily generated batches from a collection.
Python’s
[`ProcessPoolExecutor`](https://fastcore.fast.ai/parallel.html#processpoolexecutor)
is extended to allow `max_workers` to be set to `0`, to easily turn off
parallel processing. This makes it easy to debug your code in serial,
then run it in parallel. It also allows you to pass arguments to your
parallel function, and to ensure there’s a pause between calls, in case
the process you are running has race conditions.
[`parallel`](https://fastcore.fast.ai/parallel.html#parallel) makes
parallel processing even easier to use, and even adds an optional
progress bar.
### L
Like most languages, Python allows for very concise syntax for some very
common types, such as `list`, which can be constructed with `[1,2,3]`.
Perl’s designer Larry Wall explained the reasoning for this kind of
syntax:
> In metaphorical honor of Huffman’s compression code that assigns
> smaller numbers of bits to more common bytes. In terms of syntax, it
> simply means that commonly used things should be shorter, but you
> shouldn’t waste short sequences on less common constructs.
On this basis, `fastcore` has just one type that has a single letter
name: [`L`](https://fastcore.fast.ai/foundation.html#l). The reason for
this is that it is designed to be a replacement for `list`, so we want
it to be just as easy to use as `[1,2,3]`. Here’s how to create that as
an [`L`](https://fastcore.fast.ai/foundation.html#l):
``` python
L(1,2,3)
```
[1, 2, 3]
The first thing to notice is that an
[`L`](https://fastcore.fast.ai/foundation.html#l) object includes in its
representation its number of elements; that’s the `(#3)` in the output
above. If there’s more than 10 elements, it will automatically truncate
the list:
``` python
p = L.range(20).shuffle()
p
```
[14, 19, 13, 11, 8, 9, 0, 6, 4, 15, 1, 17, 16, 5, 3, 10, 2, 7, 12, 18]
[`L`](https://fastcore.fast.ai/foundation.html#l) contains many of the
same indexing ideas that NumPy’s `array` does, including indexing with a
list of indexes, or a boolean mask list:
``` python
p[2,4,6]
```
[13, 8, 0]
It also contains other methods used in `array`, such as
[`L.argwhere`](https://fastcore.fast.ai/foundation.html#l.argwhere):
``` python
p.argwhere(ge(15))
```
[1, 9, 11, 12, 19]
As you can see from this example, `fastcore` also includes a number of
features that make a functional style of programming easier, such as a
full range of boolean functions (e.g `ge`, `gt`, etc) which give the
same answer as the functions from Python’s `operator` module if given
two parameters, but return a [curried
function](https://en.wikipedia.org/wiki/Currying) if given one
parameter.
There’s too much functionality to show it all here, so be sure to check
the docs. Many little things are added that we thought should have been
in `list` in the first place, such as making this do what you’d expect
(which is an error with `list`, but works fine with
[`L`](https://fastcore.fast.ai/foundation.html#l)):
``` python
1 + L(2,3,4)
```
[1, 2, 3, 4]
---
# Source: https://fastcore.fast.ai/xdg.html.md
# XDG
See the [XDG Base Directory
Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html)
for more information.
## Overview
[`xdg_cache_home`](https://fastcore.fast.ai/xdg.html#xdg_cache_home),
[`xdg_config_home`](https://fastcore.fast.ai/xdg.html#xdg_config_home),
[`xdg_data_home`](https://fastcore.fast.ai/xdg.html#xdg_data_home), and
[`xdg_state_home`](https://fastcore.fast.ai/xdg.html#xdg_state_home)
return `pathlib.Path` objects containing the value of the environment
variable named `XDG_CACHE_HOME`, `XDG_CONFIG_HOME`, `XDG_DATA_HOME`, and
`XDG_STATE_HOME` respectively, or the default defined in the
specification if the environment variable is unset, empty, or contains a
relative path rather than absolute path.
[`xdg_config_dirs`](https://fastcore.fast.ai/xdg.html#xdg_config_dirs)
and [`xdg_data_dirs`](https://fastcore.fast.ai/xdg.html#xdg_data_dirs)
return a list of `pathlib.Path` objects containing the value, split on
colons, of the environment variable named `XDG_CONFIG_DIRS` and
`XDG_DATA_DIRS` respectively, or the default defined in the
specification if the environment variable is unset or empty. Relative
paths are ignored, as per the specification.
[`xdg_runtime_dir`](https://fastcore.fast.ai/xdg.html#xdg_runtime_dir)
returns a `pathlib.Path` object containing the value of the
`XDG_RUNTIME_DIR` environment variable, or `None` if the environment
variable is not set, or contains a relative path rather than absolute
path.
## Helpers
We’ll start by defining a context manager that temporarily sets an
environment variable to demonstrate the behaviour of each helper
function:
``` python
from contextlib import contextmanager
```
``` python
@contextmanager
def env(variable, value):
old = os.environ.get(variable, None)
try:
os.environ[variable] = value
yield
finally:
if old is None: del os.environ[variable]
else: os.environ[variable] = old
```
------------------------------------------------------------------------
source
### xdg_cache_home
``` python
def xdg_cache_home(
):
```
*Path corresponding to `XDG_CACHE_HOME`*
``` python
from fastcore.test import *
```
``` python
test_eq(xdg_cache_home(), Path.home()/'.cache')
with env('XDG_CACHE_HOME', '/home/fastai/.cache'):
test_eq(xdg_cache_home(), Path('/home/fastai/.cache'))
```
------------------------------------------------------------------------
source
### xdg_config_dirs
``` python
def xdg_config_dirs(
):
```
*Paths corresponding to `XDG_CONFIG_DIRS`*
``` python
test_eq(xdg_config_dirs(), [Path('/etc/xdg')])
with env('XDG_CONFIG_DIRS', '/home/fastai/.xdg:/home/fastai/.config'):
test_eq(xdg_config_dirs(), [Path('/home/fastai/.xdg'), Path('/home/fastai/.config')])
```
------------------------------------------------------------------------
source
### xdg_config_home
``` python
def xdg_config_home(
):
```
*Path corresponding to `XDG_CONFIG_HOME`*
``` python
test_eq(xdg_config_home(), Path.home()/'.config')
with env('XDG_CONFIG_HOME', '/home/fastai/.config'):
test_eq(xdg_config_home(), Path('/home/fastai/.config'))
```
------------------------------------------------------------------------
source
### xdg_data_dirs
``` python
def xdg_data_dirs(
):
```
*Paths corresponding to XDG_DATA_DIRS\`*
------------------------------------------------------------------------
source
### xdg_data_home
``` python
def xdg_data_home(
):
```
*Path corresponding to `XDG_DATA_HOME`*
``` python
test_eq(xdg_data_home(), Path.home()/'.local/share')
with env('XDG_DATA_HOME', '/home/fastai/.data'):
test_eq(xdg_data_home(), Path('/home/fastai/.data'))
```
------------------------------------------------------------------------
source
### xdg_runtime_dir
``` python
def xdg_runtime_dir(
):
```
*Path corresponding to `XDG_RUNTIME_DIR`*
------------------------------------------------------------------------
source
### xdg_state_home
``` python
def xdg_state_home(
):
```
*Path corresponding to `XDG_STATE_HOME`*
``` python
test_eq(xdg_state_home(), Path.home()/'.local/state')
with env('XDG_STATE_HOME', '/home/fastai/.state'):
test_eq(xdg_state_home(), Path('/home/fastai/.state'))
```
------------------------------------------------------------------------
Copyright © 2016-2021 Scott Stevenson <script>alert('XSS')</script>
\n"
``` python
h = to_xml(samp, do_escape=False)
print(h)
```
Some text
another line
``` python
c = I('hello')
print(c)
```
hello
``` python
c
```
hello
``` python
class PageTitle:
def __ft__(self): return H1("Hello")
class HomePage:
def __ft__(self): return Div(PageTitle(), Div('hello'))
h = to_xml(Div(HomePage()))
expected_output = """Hello
hello
Hello
hello
Some text
another line
Interoperability both directions with Django and Jinja using the
[**html**()
protocol](https://jinja.palletsprojects.com/en/3.1.x/templates/#jinja-filters.escape):
``` python
def _esc(s): return s.__html__() if hasattr(s, '__html__') else Safe(escape(s))
r = Safe('Hello from Django')
print(to_xml(Div(r)))
print(_esc(Div(P('Hello from fastcore <3'))))
```
Hello from Django
Hello from fastcore <3
hi
FT components also stringify with [`to_xml`](https://fastcore.fast.ai/xml.html#to_xml): ``` python print(Div('ho')) ```ho
------------------------------------------------------------------------
source
### FT.\_\_hash\_\_
``` python
def __hash__(
):
```
*Return hash(self).*
------------------------------------------------------------------------
source
### FT.\_\_eq\_\_
``` python
def __eq__(
other
):
```
*Return self==value.*
[`FT`](https://fastcore.fast.ai/xml.html#ft) object equality and hashing
is based on tag, attrs, and children.
``` python
test_eq(Div('hello', id='x'), Div('hello', id='x'))
test_ne(Div('hello'), Div('goodbye'))
test_ne(Div('hello', id='a'), Div('hello', id='b'))
test_ne(P('hello'), Div('hello'))
test_eq(hash(Div('hello', id='x')), hash(Div('hello', id='x')))
assert hash(Div('hello')), hash(Div('goodbye'))
```
## Display
------------------------------------------------------------------------
source
### highlight
``` python
def highlight(
s, lang:str='html'
):
```
*Markdown to syntax-highlight `s` in language `lang`*
------------------------------------------------------------------------
source
### showtags
``` python
def showtags(
s
):
```
You can also reorder the children to come *after* the attrs, if you use
this alternative syntax for [`FT`](https://fastcore.fast.ai/xml.html#ft)
where the children are in a second pair of `()` (behind the scenes this
is because [`FT`](https://fastcore.fast.ai/xml.html#ft) implements
`__call__` to add children).
``` python
hl_md(
Body(klass='myclass')(
Div(style='padding:3px')(
'Some text 1<2',
I(spurious=True)('in italics'),
Input(name='me'),
Img(src="filename", data=1)
)
))
```
``` html
Some text 1<2in italics
```
------------------------------------------------------------------------
source
### **getattr**
``` python
def __getattr__(
tag
):
```
---
# Source: https://fastcore.fast.ai/xtras.html.md
# Utility functions
## File Functions
Utilities (other than extensions to Pathlib.Path) for dealing with IO.
------------------------------------------------------------------------
source
### walk
``` python
def walk(
path:Path | str, # path to start searching
symlinks:bool=True, # follow symlinks?
keep_file:callable=ret_true, # function that returns True for wanted files
keep_folder:callable=ret_true, # function that returns True for folders to enter
skip_folder:callable=ret_false, # function that returns True for folders to skip
func:callable=join, # function to apply to each matched file
ret_folders:bool=False, # return folders, not just files
sort:bool=True, # sort files by name within each folder
):
```
*Generator version of `os.walk`, using functions to filter files and
folders*
------------------------------------------------------------------------
source
### exttypes
``` python
def exttypes(
types
):
```
*Get exts for comma-separated or list `typ`; if not found in list,
return list with just `types`.* Supported: py, js, java, c, cpp, rb, r,
ex, sh, web, doc, cfg
``` python
print(exttypes('py,doc'))
print(exttypes('zig,txt'))
```
['ipynb', 'py', 'md', 'rst']
['zig', 'txt']
------------------------------------------------------------------------
source
### globtastic
``` python
def globtastic(
path:Path | str='.', # path to start searching
recursive:bool=True, # search subfolders
symlinks:bool=True, # follow symlinks?
file_glob:str=None, # Only include files matching glob
file_re:str=None, # Only include files matching regex
folder_re:str=None, # Only enter folders matching regex
skip_file_glob:str=None, # Skip files matching glob
skip_file_re:str=None, # Skip files matching regex
skip_folder_re:str=None, # Skip folders matching regex,
func:callable=join, # function to apply to each matched file
ret_folders:bool=False, # return folders, not just files
sort:bool=True, # sort files by name within each folder
types:str | list=None, # list or comma-separated str of ext types from: py, js, java, c, cpp, rb, r, ex, sh, web, doc, cfg
exts:str | list=None, # list or comma-separated str of exts to include
)->L: # Paths to matched files
```
*A more powerful `glob`, including regex matches, symlink handling, and
skip parameters*
``` python
globtastic('.', skip_folder_re='^[_.]', folder_re='core', file_glob='*.*py*', file_re='c')
```
['./fastcore/basics.py', './fastcore/dispatch.py', './fastcore/docments.py', './fastcore/docscrape.py', './fastcore/script.py']
``` python
globtastic(skip_folder_re='^[_.]', folder_re='core', types='py', file_re='c', skip_file_re='^_', sort=True)
```
['./fastcore/all.py', './fastcore/ansi.py', './fastcore/basics.py', './fastcore/dispatch.py', './fastcore/docments.py', './fastcore/docscrape.py', './fastcore/foundation.py', './fastcore/imghdr.py', './fastcore/imports.py', './fastcore/meta.py', './fastcore/nb_imports.py', './fastcore/net.py', './fastcore/parallel.py', './fastcore/py2pyi.py', './fastcore/script.py', './fastcore/shutil.py', './fastcore/style.py', './fastcore/test.py', './fastcore/tools.py', './fastcore/transform.py', './fastcore/utils.py', './fastcore/xdg.py', './fastcore/xml.py', './fastcore/xtras.py']
------------------------------------------------------------------------
source
### pglob
``` python
def pglob(
path:Path | str='.', # path to start searching
func:callable=Path, # function to apply to each matched file
recursive:bool=True, symlinks:bool=True, file_glob:str=None, file_re:str=None, folder_re:str=None,
skip_file_glob:str=None, skip_file_re:str=None, skip_folder_re:str=None, ret_folders:bool=False, sort:bool=True,
types:str | list=None, exts:str | list=None
)->L: # Paths to matched files
```
*Shortcut for `globtastic(..., call=Path)`*
``` python
pglob('..', skip_folder_re='^[_.]', types='doc', skip_file_re='^_')[:6]
```
[Path('../CHANGELOG.md'), Path('../CODE_OF_CONDUCT.md'), Path('../CONTRIBUTING.md'), Path('../README.md')]
------------------------------------------------------------------------
source
### maybe_open
``` python
def maybe_open(
f, mode:str='r', kwargs:VAR_KEYWORD
):
```
*Context manager: open `f` if it is a path (and close on exit)*
This is useful for functions where you want to accept a path *or* file.
[`maybe_open`](https://fastcore.fast.ai/xtras.html#maybe_open) will not
close your file handle if you pass one in.
``` python
def _f(fn):
with maybe_open(fn) as f: return f.encoding
fname = '00_test.ipynb'
sys_encoding = 'cp1252' if sys.platform == 'win32' else 'utf-8'
test_eq(_f(fname).lower(), sys_encoding)
with open(fname) as fh: test_eq(_f(fh).lower(), sys_encoding)
```
For example, we can use this to reimplement
[`imghdr.what`](https://docs.python.org/3/library/imghdr.html#imghdr.what)
from the Python standard library, which is [written in Python
3.9](https://github.com/python/cpython/blob/3.9/Lib/imghdr.py#L11) as:
``` python
from fastcore import imghdr
```
``` python
def what(file, h=None):
f = None
try:
if h is None:
if isinstance(file, (str,os.PathLike)):
f = open(file, 'rb')
h = f.read(32)
else:
location = file.tell()
h = file.read(32)
file.seek(location)
for tf in imghdr.tests:
res = tf(h, f)
if res: return res
finally:
if f: f.close()
return None
```
Here’s an example of the use of this function:
``` python
fname = 'images/puppy.jpg'
what(fname)
```
'jpeg'
With [`maybe_open`](https://fastcore.fast.ai/xtras.html#maybe_open),
`Self`, and
[`L.map_first`](https://fastcore.fast.ai/foundation.html#l.map_first),
we can rewrite this in a much more concise and (in our opinion) clear
way:
``` python
def what(file, h=None):
if h is None:
with maybe_open(file, 'rb') as f: h = f.peek(32)
return L(imghdr.tests).map_first(Self(h,file))
```
…and we can check that it still works:
``` python
test_eq(what(fname), 'jpeg')
```
…along with the version passing a file handle:
``` python
with open(fname,'rb') as f: test_eq(what(f), 'jpeg')
```
…along with the `h` parameter version:
``` python
with open(fname,'rb') as f: test_eq(what(None, h=f.read(32)), 'jpeg')
```
------------------------------------------------------------------------
source
### mkdir
``` python
def mkdir(
path, exist_ok:bool=False, parents:bool=False, overwrite:bool=False, kwargs:VAR_KEYWORD
):
```
*Creates and returns a directory defined by `path`, optionally removing
previous existing directory if `overwrite` is `True`*
``` python
with tempfile.TemporaryDirectory() as d:
path = Path(os.path.join(d, 'new_dir'))
new_dir = mkdir(path)
assert new_dir.exists()
test_eq(new_dir, path)
# test overwrite
with open(new_dir/'test.txt', 'w') as f: f.writelines('test')
test_eq(len(list(walk(new_dir))), 1) # assert file is present
new_dir = mkdir(new_dir, overwrite=True)
test_eq(len(list(walk(new_dir))), 0) # assert file was deleted
```
------------------------------------------------------------------------
source
### image_size
``` python
def image_size(
fn
):
```
*Tuple of (w,h) for png, gif, or jpg; `None` otherwise*
``` python
test_eq(image_size(fname), (1200,803))
```
``` python
from PIL import Image
from IPython.display import Image as IPImage
```
``` python
img = Image.new('RGB', (50, 50), color='red')
img
```
------------------------------------------------------------------------
source
### img_bytes
``` python
def img_bytes(
img, fmt:str='PNG'
):
```
``` python
ib = img_bytes(img)
IPImage(ib)
```
------------------------------------------------------------------------
source
### detect_mime
``` python
def detect_mime(
data
):
```
*Get the MIME type for bytes `data`, covering common PDF, audio, video,
and image types*
``` python
detect_mime(ib)
```
'image/png'
``` python
test_is(detect_mime('this is a non-byte string'),None)
```
------------------------------------------------------------------------
source
### bunzip
``` python
def bunzip(
fn
):
```
*bunzip `fn`, raising exception if output already exists*
``` python
f = Path('files/test.txt')
if f.exists(): f.unlink()
bunzip('files/test.txt.bz2')
t = f.open().readlines()
test_eq(len(t),1)
test_eq(t[0], 'test\n')
f.unlink()
```
------------------------------------------------------------------------
source
### loads
``` python
def loads(
s, kw:VAR_KEYWORD
):
```
*Same as `json.loads`, but handles `None`*
------------------------------------------------------------------------
source
### loads_multi
``` python
def loads_multi(
s:str
):
```
*Generator of \>=0 decoded json dicts, possibly with non-json ignored
text at start and end*
``` python
tst = """
# ignored
{ "a":1 }
hello
{
"b":2
}
"""
test_eq(list(loads_multi(tst)), [{'a': 1}, {'b': 2}])
```
------------------------------------------------------------------------
source
### dumps
``` python
def dumps(
obj, kw:VAR_KEYWORD
):
```
*Same as `json.dumps`, but uses `ujson` if available*
------------------------------------------------------------------------
source
### untar_dir
``` python
def untar_dir(
fname, dest, rename:bool=False, overwrite:bool=False
):
```
*untar `file` into `dest`, creating a directory if the root contains
more than one item*
``` python
def test_untar(foldername, rename=False, **kwargs):
with tempfile.TemporaryDirectory() as d:
nm = os.path.join(d, 'a')
shutil.make_archive(nm, 'gztar', **kwargs)
with tempfile.TemporaryDirectory() as d2:
d2 = Path(d2)
untar_dir(nm+'.tar.gz', d2, rename=rename)
test_eq(d2.ls(), [d2/foldername])
```
If the contents of `fname` contain just one file or directory, it is
placed directly in `dest`:
``` python
# using `base_dir` in `make_archive` results in `images` directory included in file names
test_untar('images', base_dir='images')
```
If `rename` then the directory created is named based on the archive,
without extension:
``` python
test_untar('a', base_dir='images', rename=True)
```
If the contents of `fname` contain multiple files and directories, a new
folder in `dest` is created with the same name as `fname` (but without
extension):
``` python
# using `root_dir` in `make_archive` results in `images` directory *not* included in file names
test_untar('a', root_dir='images')
```
------------------------------------------------------------------------
source
### repo_details
``` python
def repo_details(
url
):
```
*Tuple of `owner,name` from ssh or https git repo `url`*
``` python
test_eq(repo_details('https://github.com/fastai/fastai.git'), ['fastai', 'fastai'])
test_eq(repo_details('git@github.com:fastai/nbdev.git\n'), ['fastai', 'nbdev'])
```
------------------------------------------------------------------------
source
### shell
``` python
def shell(
args:VAR_POSITIONAL, kwargs:VAR_KEYWORD
):
```
*Shortcut for `subprocess.run(shell=True)`*
------------------------------------------------------------------------
source
### ssh
``` python
def ssh(
host, args:str='', user:str='ubuntu', sock:NoneType=None
):
```
*Run SSH command with given arguments*
------------------------------------------------------------------------
source
### rsync_multi
``` python
def rsync_multi(
ip, files, user:str='ubuntu', persist:str='5m'
):
```
*Transfer multiple files with rename using persistent SSH connection*
------------------------------------------------------------------------
source
### run
``` python
def run(
cmd, rest:VAR_POSITIONAL, same_in_win:bool=False, ignore_ex:bool=False, as_bytes:bool=False, stderr:bool=True
):
```
*Pass `cmd` (splitting with `shlex` if string) to `subprocess.run`;
return `stdout`; raise `IOError` if fails*
You can pass a string (which will be split based on standard shell
rules), a list, or pass args directly:
``` python
run('echo', same_in_win=True)
run('pip', '--version', same_in_win=True)
run(['pip', '--version'], same_in_win=True)
```
'pip 25.3 from /Users/rensdimmendaal/git/.venv/lib/python3.12/site-packages/pip (python 3.12)'
``` python
if sys.platform == 'win32':
assert 'ipynb' in run('cmd /c dir /p')
assert 'ipynb' in run(['cmd', '/c', 'dir', '/p'])
assert 'ipynb' in run('cmd', '/c', 'dir', '/p')
else:
assert 'ipynb' in run('ls -ls')
assert 'ipynb' in run(['ls', '-l'])
assert 'ipynb' in run('ls', '-l')
```
Some commands fail in non-error situations, like `grep`. Use `ignore_ex`
in those cases, which will return a tuple of stdout and returncode:
``` python
if sys.platform == 'win32':
test_eq(run('cmd /c findstr asdfds 00_test.ipynb', ignore_ex=True)[0], 1)
else:
test_eq(run('grep asdfds 00_test.ipynb', ignore_ex=True)[0], 1)
```
[`run`](https://fastcore.fast.ai/xtras.html#run) automatically decodes
returned bytes to a `str`. Use `as_bytes` to skip that:
``` python
if sys.platform == 'win32':
test_eq(run('cmd /c echo hi'), 'hi')
else:
test_eq(run('echo hi', as_bytes=True), b'hi\n')
```
------------------------------------------------------------------------
source
### open_file
``` python
def open_file(
fn, mode:str='r', kwargs:VAR_KEYWORD
):
```
*Open a file, with optional compression if gz or bz2 suffix*
------------------------------------------------------------------------
source
### save_pickle
``` python
def save_pickle(
fn, o
):
```
*Save a pickle file, to a file name or opened file*
------------------------------------------------------------------------
source
### load_pickle
``` python
def load_pickle(
fn
):
```
*Load a pickle file from a file name or opened file*
``` python
for suf in '.pkl','.bz2','.gz':
# delete=False is added for Windows
# https://stackoverflow.com/questions/23212435/permission-denied-to-write-to-my-temporary-file
with tempfile.NamedTemporaryFile(suffix=suf, delete=False) as f:
fn = Path(f.name)
save_pickle(fn, 't')
t = load_pickle(fn)
f.close()
test_eq(t,'t')
```
------------------------------------------------------------------------
source
### parse_env
``` python
def parse_env(
s:str=None, fn:Union[str, Path]=None
)->dict:
```
*Parse a shell-style environment string or file*
``` python
testf = """# comment
# another comment
export FOO="bar#baz"
BAR=thing # comment "ok"
baz='thong'
QUX=quux
export ZAP = "zip" # more comments
FOOBAR = 42 # trailing space and comment"""
exp = dict(FOO='bar#baz', BAR='thing', baz='thong', QUX='quux', ZAP='zip', FOOBAR='42')
test_eq(parse_env(testf), exp)
```
------------------------------------------------------------------------
source
### expand_wildcards
``` python
def expand_wildcards(
code
):
```
*Expand all wildcard imports in the given code string.*
``` python
inp = """from math import *
from os import *
from random import *
def func(): return sin(pi) + path.join('a', 'b') + randint(1, 10)"""
exp = """from math import pi, sin
from os import path
from random import randint
def func(): return sin(pi) + path.join('a', 'b') + randint(1, 10)"""
test_eq(expand_wildcards(inp), exp)
inp = """from itertools import *
def func(): pass"""
test_eq(expand_wildcards(inp), inp)
inp = """def outer():
from math import *
def inner():
from os import *
return sin(pi) + path.join('a', 'b')"""
exp = """def outer():
from math import pi, sin
def inner():
from os import path
return sin(pi) + path.join('a', 'b')"""
test_eq(expand_wildcards(inp), exp)
```
## Collections
------------------------------------------------------------------------
source
### dict2obj
``` python
def dict2obj(
d:NoneType=None, list_func:_L_Meta=L, dict_func:type=AttrDict, kwargs:VAR_KEYWORD
):
```
*Convert (possibly nested) dicts (or lists of dicts) to
[`AttrDict`](https://fastcore.fast.ai/basics.html#attrdict)*
This is a convenience to give you “dotted” access to (possibly nested)
dictionaries, e.g:
``` python
d1 = dict(a=1, b=dict(c=2,d=3))
d2 = dict2obj(d1)
test_eq(d2.b.c, 2)
test_eq(d2.b['c'], 2)
```
kwargs can also be used:
``` python
d3 = dict2obj(a=1, b=dict(c=2,d=3))
test_eq(d3.b.c, 2)
test_eq(d3.b['c'], 2)
```
It can also be used on lists of dicts.
``` python
_list_of_dicts = [d1, d1]
ds = dict2obj(_list_of_dicts)
test_eq(ds[0].b.c, 2)
```
------------------------------------------------------------------------
source
### obj2dict
``` python
def obj2dict(
d
):
```
*Convert (possibly nested) AttrDicts (or lists of AttrDicts) to `dict`*
[`obj2dict`](https://fastcore.fast.ai/xtras.html#obj2dict) can be used
to reverse what is done by
[`dict2obj`](https://fastcore.fast.ai/xtras.html#dict2obj):
``` python
test_eq(obj2dict(d2), d1)
test_eq(obj2dict(ds), _list_of_dicts)
```
------------------------------------------------------------------------
source
### repr_dict
``` python
def repr_dict(
d
):
```
*Print nested dicts and lists, such as returned by
[`dict2obj`](https://fastcore.fast.ai/xtras.html#dict2obj)*
``` python
print(repr_dict(d2))
```
- a: 1
- b:
- c: 2
- d: 3
------------------------------------------------------------------------
source
### is_listy
``` python
def is_listy(
x
):
```
*`isinstance(x, (tuple,list,L,slice,Generator))`*
``` python
assert is_listy((1,))
assert is_listy([1])
assert is_listy(L([1]))
assert is_listy(slice(2))
assert not is_listy(array([1]))
```
------------------------------------------------------------------------
source
### mapped
``` python
def mapped(
f, it
):
```
*map `f` over `it`, unless it’s not listy, in which case return `f(it)`*
``` python
def _f(x,a=1): return x-a
test_eq(mapped(_f,1),0)
test_eq(mapped(_f,[1,2]),[0,1])
test_eq(mapped(_f,(1,)),(0,))
```
## Extensions to Pathlib.Path
The following methods are added to the standard python libary
[Pathlib.Path](https://docs.python.org/3/library/pathlib.html#basic-use).
------------------------------------------------------------------------
source
### Path.readlines
``` python
def readlines(
hint:int=-1, encoding:str='utf8'
):
```
*Read the content of `self`*
------------------------------------------------------------------------
source
### Path.read_json
``` python
def read_json(
encoding:NoneType=None, errors:NoneType=None
):
```
*Same as `read_text` followed by
[`loads`](https://fastcore.fast.ai/xtras.html#loads)*
------------------------------------------------------------------------
source
### Path.mk_write
``` python
def mk_write(
data, encoding:NoneType=None, errors:NoneType=None, mode:int=511, uid:int=-1, gid:int=-1
):
```
*Make all parent dirs of `self`, and write `data`*
------------------------------------------------------------------------
source
### Path.write_json
``` python
def write_json(
data, encoding:NoneType=None, errors:NoneType=None, mode:int=511, uid:int=-1, gid:int=-1, kw:VAR_KEYWORD
):
```
*Same as [`dumps`](https://fastcore.fast.ai/xtras.html#dumps)followed by
`mk_write`*
------------------------------------------------------------------------
source
### Path.relpath
``` python
def relpath(
start:NoneType=None
):
```
*Same as `os.path.relpath`, but returns a `Path`, and resolves symlinks*
``` python
p = Path('../fastcore/').resolve()
p
```
Path('/Users/rensdimmendaal/git/repos/fastcore/fastcore')
``` python
p.relpath(Path.cwd())
```
Path('../fastcore')
------------------------------------------------------------------------
source
### Path.ls
``` python
def ls(
n_max:NoneType=None, file_type:NoneType=None, file_exts:NoneType=None
):
```
*Contents of path as a list*
We add an `ls()` method to `pathlib.Path` which is simply defined as
`list(Path.iterdir())`, mainly for convenience in REPL environments such
as notebooks.
``` python
path = Path()
t = path.ls()
assert len(t)>0
t1 = path.ls(10)
test_eq(len(t1), 10)
t2 = path.ls(file_exts='.ipynb')
assert len(t)>len(t2)
t[0]
```
Path('llms.txt')
You can also pass an optional `file_type` MIME prefix and/or a list of
file extensions.
``` python
lib_path = (path/'../fastcore')
txt_files=lib_path.ls(file_type='text')
assert len(txt_files) > 0 and txt_files[0].suffix=='.py'
ipy_files=path.ls(file_exts=['.ipynb'])
assert len(ipy_files) > 0 and ipy_files[0].suffix=='.ipynb'
txt_files[0],ipy_files[0]
```
(Path('../fastcore/shutil.py'), Path('000_tour.ipynb'))
------------------------------------------------------------------------
source
### Path.normpath
``` python
def normpath(
):
```
*Normalize path, eliminating double slashes, etc.*
`normpath` normalizes a path by collapsing redundant separators and
up-level references (e.g., `..`).
``` python
p = Path('foo//bar/../baz')
p.normpath()
```
Path('foo/baz')
------------------------------------------------------------------------
source
### Path.\_\_repr\_\_
``` python
def __repr__(
):
```
*Return repr(self).*
fastai also updates the `repr` of `Path` such that, if `Path.BASE_PATH`
is defined, all paths are printed relative to that path (as long as they
are contained in `Path.BASE_PATH`:
``` python
t = ipy_files[0].absolute()
try:
Path.BASE_PATH = t.parent.parent
test_eq(repr(t), f"Path('nbs/{t.name}')")
finally: Path.BASE_PATH = None
```
------------------------------------------------------------------------
source
### Path.delete
``` python
def delete(
):
```
*Delete a file, symlink, or directory tree*
## Reindexing Collections
------------------------------------------------------------------------
source
#### ReindexCollection
``` python
def ReindexCollection(
coll, idxs:NoneType=None, cache:NoneType=None, tfm:function=noop
):
```
*Reindexes collection `coll` with indices `idxs` and optional LRU cache
of size `cache`*
This is useful when constructing batches or organizing data in a
particular manner (i.e. for deep learning). This class is primarly used
in organizing data for language models in fastai.
You can supply a custom index upon instantiation with the `idxs`
argument, or you can call the `reindex` method to supply a new index for
your collection.
Here is how you can reindex a list such that the elements are reversed:
``` python
rc=ReindexCollection(['a', 'b', 'c', 'd', 'e'], idxs=[4,3,2,1,0])
list(rc)
```
['e', 'd', 'c', 'b', 'a']
Alternatively, you can use the `reindex` method:
------------------------------------------------------------------------
source
###### ReindexCollection.reindex
``` python
def reindex(
idxs
):
```
*Replace `self.idxs` with idxs*
``` python
rc=ReindexCollection(['a', 'b', 'c', 'd', 'e'])
rc.reindex([4,3,2,1,0])
list(rc)
```
['e', 'd', 'c', 'b', 'a']
You can optionally specify a LRU cache, which uses
[functools.lru_cache](https://docs.python.org/3/library/functools.html#functools.lru_cache)
upon instantiation:
``` python
sz = 50
t = ReindexCollection(L.range(sz), cache=2)
#trigger a cache hit by indexing into the same element multiple times
t[0], t[0]
t._get.cache_info()
```
CacheInfo(hits=1, misses=1, maxsize=2, currsize=1)
You can optionally clear the LRU cache by calling the `cache_clear`
method:
------------------------------------------------------------------------
source
##### ReindexCollection.cache_clear
``` python
def cache_clear(
):
```
*Clear LRU cache*
``` python
sz = 50
t = ReindexCollection(L.range(sz), cache=2)
#trigger a cache hit by indexing into the same element multiple times
t[0], t[0]
t.cache_clear()
t._get.cache_info()
```
CacheInfo(hits=0, misses=0, maxsize=2, currsize=0)
------------------------------------------------------------------------
source
##### ReindexCollection.shuffle
``` python
def shuffle(
):
```
*Randomly shuffle indices*
Note that an ordered index is automatically constructed for the data
structure even if one is not supplied.
``` python
rc=ReindexCollection(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'])
rc.shuffle()
list(rc)
```
['a', 'g', 'h', 'c', 'd', 'b', 'f', 'e']
``` python
sz = 50
t = ReindexCollection(L.range(sz), cache=2)
test_eq(list(t), range(sz))
test_eq(t[sz-1], sz-1)
test_eq(t._get.cache_info().hits, 1)
t.shuffle()
test_eq(t._get.cache_info().hits, 1)
test_ne(list(t), range(sz))
test_eq(set(t), set(range(sz)))
t.cache_clear()
test_eq(t._get.cache_info().hits, 0)
test_eq(t.count(0), 1)
```
## [`SaveReturn`](https://fastcore.fast.ai/xtras.html#savereturn) and [`save_iter`](https://fastcore.fast.ai/xtras.html#save_iter) Variants
These utilities solve a common problem in Python: how to extract
additional information from generator functions beyond just the yielded
values.
In Python, generator functions can `yield` values and also `return` a
final value, but the return value is normally lost when you iterate over
the generator:
``` python
def example_generator():
total = 0
for i in range(3):
total += i
yield i
return total # This gets lost!
# The return value (3) is lost
values = list(example_generator()) # [0, 1, 2]
```
------------------------------------------------------------------------
source
### SaveReturn
``` python
def SaveReturn(
its
):
```
*Wrap an iterator such that the generator function’s return value is
stored in `.value`*