Merge branch 'micropython:master' into master

Author: wemos

.github/workflows/build_packages.yml

@@ -0,0 +1,27 @@
+name: Build all packages
+
+on: [push, pull_request]
+
+env:
+  PACKAGE_INDEX_PATH: /tmp/micropython-lib-deploy
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - uses: actions/setup-python@v1
+    - name: Setup environment
+      run: source tools/ci.sh && ci_build_packages_setup
+    - name: Check manifest files
+      run: source tools/ci.sh && ci_build_packages_check_manifest
+    - name: Compile package index
+      run: source tools/ci.sh && ci_build_packages_compile_index
+    - name: Publish packages for branch
+      if: vars.MICROPY_PUBLISH_MIP_INDEX && github.event_name == 'push' && ! github.event.deleted
+      run: source tools/ci.sh && ci_push_package_index
+    - name: Upload packages as artifact
+      uses: actions/upload-artifact@v3
+      with:
+        name: packages-${{ github.sha }}
+        path: ${{ env.PACKAGE_INDEX_PATH }}

.github/workflows/cleanup_published_packages.yml

@@ -0,0 +1,12 @@
+name: Cleanup published packages
+
+on: delete
+
+jobs:
+  cleanup:
+    runs-on: ubuntu-latest
+    if: vars.MICROPY_PUBLISH_MIP_INDEX
+    steps:
+    - uses: actions/checkout@v2
+    - name: Clean up published files
+      run: source tools/ci.sh && ci_cleanup_package_index ${{ github.event.ref }}

CODEOFCONDUCT.md

@@ -0,0 +1 @@
+Please see the [MicroPython Code of Conduct](https://github.com/micropython/micropython/blob/master/CODEOFCONDUCT.md).

CONTRIBUTING.md

@@ -1,3 +1,117 @@
-If you submit a pull request, please adhere to Contributor Guidelines:
+## Contributor's Guidelines & Code Conventions
 
-https://github.com/micropython/micropython-lib/wiki/ContributorGuidelines
+micropython-lib follows the same general conventions as the [main MicroPython
+repository](https://github.com/micropython/micropython). Please see
+[micropython/CONTRIBUTING.md](https://github.com/micropython/micropython/blob/master/CONTRIBUTING.md)
+and [micropython/CODECONVENTIONS.md](https://github.com/micropython/micropython/blob/master/CODECONVENTIONS.md).
+
+### Raising issues
+
+Please include enough information for someone to reproduce the issue you are
+describing. This will typically include:
+
+* The version of MicroPython you are using (e.g. the firmware filename, git
+  hash, or version info printed by the startup message).
+* What board/device you are running MicroPython on.
+* Which package you have installed, how you installed it, and what version.
+  When installed via `mip`, all packages will have a `__version__`
+  attribute.
+* A simple code snippet that demonstrates the issue.
+
+If you have a how-to question or are looking for help with using MicroPython
+or packages from micropython-lib, please post at the
+[discussion forum](https://github.com/orgs/micropython/discussions) instead.
+
+### Pull requests
+
+The same rules for commit messages, signing-off commits, and commit structure
+apply as for the main MicroPython repository. All Python code is formatted
+using `black`. See [`tools/codeformat.py`](tools/codeformat.py) to apply
+`black` automatically before submitting a PR.
+
+There are some specific conventions and guidelines for micropython-lib:
+
+* The first line of the commit message should start with the name of the
+  package, followed by a short description of the commit. Package names are
+  globally unique in the micropython-lib directory structure.
+
+  For example: `shutil: Add disk_usage function.`
+
+* Although we encourage keeping the code short and minimal, please still use
+  comments in your code. Typically, packages will be installed via
+  `mip` and so they will be compiled to bytecode where comments will
+  _not_ contribute to the installed size.
+
+* All packages must include a `manifest.py`, including a `metadata()` line
+  with at least a description and a version.
+
+* Prefer to break larger packages up into smaller chunks, so that just the
+  required functionality can be installed. The way to do this is to have a
+  base package, e.g. `mypackage` containing `mypackage/__init__.py`, and then
+  an "extension" package, e.g. `mypackage-ext` containing additional files
+  e.g. `mypackage/ext.py`. See
+  [`collections-defaultdict`](python-stdlib/collections-defaultdict) as an
+  example.
+
+* If you think a package might be extended in this way in the future, prefer
+  to create a package directory with `package/__init__.py`, rather than a
+  single `module.py`.
+
+* Packages in the python-stdlib directory should be CPython compatible and
+  implement a subset of the CPython equivalent. Avoid adding
+  MicroPython-specific extensions. Please include a link to the corresponding
+  CPython docs in the PR.
+
+* Include tests (ideally using the `unittest` package) as `test_*.py`.
+  Otherwise, provide examples as `example_*.py`. When porting CPython
+  packages, prefer to use the existing tests rather than writing new ones
+  from scratch.
+
+* When porting an existing third-party package, please ensure that the source
+  license is compatible.
+
+* To make it easier for others to install packages directly from your PR before
+  it is merged, consider opting-in to automatic package publishing (see
+  [Publishing packages from forks](#publishing-packages-from-forks)). If you do
+  this, consider quoting the [commands to install
+  packages](README.md#installing-packages-from-forks) in your Pull Request
+  description.
+
+### Publishing packages from forks
+
+You can easily publish the packages from your micropython-lib
+[fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/about-forks)
+by opting in to a system based on [GitHub
+Actions](https://docs.github.com/en/actions) and [GitHub
+Pages](https://docs.github.com/en/pages):
+
+1. Open your fork's repository in the GitHub web interface.
+2. Navigate to "Settings" -> "Secrets and variables" -> "Actions" -> "Variables".
+3. Click "New repository variable"
+4. Create a variable named `MICROPY_PUBLISH_MIP_INDEX` with value `true` (or any
+   "truthy" value).
+5. The settings for GitHub Actions and GitHub Pages features should not need to
+   be changed from the repository defaults, unless you've explicitly disabled
+   them.
+
+The next time you push commits to a branch in your fork, GitHub Actions will run
+an additional step in the "Build All Packages" workflow named "Publish Packages
+for branch".
+
+Anyone can then install these packages as described under [Installing packages
+from forks](README.md#installing-packages-from-forks). The exact commands are also
+quoted in the GitHub Actions log for the "Publish Packages for branch" step.
+
+#### Opting Back Out
+
+To opt-out again, delete the `MICROPY_PUBLISH_MIP_INDEX` variable and
+(optionally) delete the `gh-pages` branch from your fork.
+
+*Note*: While enabled, all micropython-lib packages will be published each time
+a change is pushed to any branch in your fork. A commit is added to the
+`gh-pages` branch each time. In a busy repository, the `gh-pages` branch may
+become quite large. The actual `.git` directory size on disk should still be
+quite small, as most of the content will be duplicated. If you're worried that
+the `gh-pages` branch has become too large then you can always delete this
+branch from GitHub. GitHub Actions will create a new `gh-pages` branch the next
+time you push a change.

README.md

@@ -1,31 +1,172 @@
-micropython-lib
-===============
+# micropython-lib
 
-This is a repository of libraries designed to be useful for writing
-MicroPython applications.
+This is a repository of packages designed to be useful for writing MicroPython
+applications.
 
-The libraries here fall into four categories corresponding to the four top-level directories:
+The packages here fall into categories corresponding to the four top-level
+directories:
 
- * **python-stdlib**: Compatible versions of modules from the [Python Standard Library](https://docs.python.org/3/library/). These should be drop-in replacements for the Python libraries, although many have reduced functionality or missing methods or classes (which may not be an issue for many most cases).
+* **python-stdlib**: Compatible versions of modules from [The Python Standard
+    Library](https://docs.python.org/3/library/). These should be drop-in
+    replacements for the corresponding Python modules, although many have
+    reduced functionality or missing methods or classes (which may not be an
+    issue for most cases).
 
- * **python-ecosys**: Compatible, but reduced-functionality versions of modules from the larger Python ecosystem, for example that might be found in the [Python Package Index](https://pypi.org/).
+ * **python-ecosys**: Compatible, but reduced-functionality versions of
+     packages from the wider Python ecosystem. For example, a package that
+     might be found in the [Python Package Index](https://pypi.org/).
 
-* **micropython**: MicroPython-specific modules that do not have equivalents in other Python environments. These are typically hardware drivers or highly-optimised alternative implementations of functionality available in other Python modules.
+ * **micropython**: MicroPython-specific packages that do not have equivalents
+     in other Python environments. This includes drivers for hardware
+     (e.g. sensors, peripherals, or displays), libraries to work with
+     embedded functionality (e.g. bluetooth), or MicroPython-specific
+     packages that do not have equivalents in CPython.
 
- * **unix-ffi**: These modules are specifically for the MicroPython Unix port and provide access to operating-system and third-party libraries via FFI.
+* **unix-ffi**: These packages are specifically for the MicroPython Unix port
+    and provide access to operating-system and third-party libraries via FFI,
+    or functionality that is not useful for non-Unix ports.
 
-Usage
------
+## Usage
 
-Many libraries are self contained modules, and you can quickly get started by
-copying the relevant Python file to your device. For example, to add the
-`base64` library, you can directly copy `python-stdlib/base64/base64.py` to the `lib`
-directory on your device.
+To install a micropython-lib package, there are four main options. For more
+information see the [Package management documentation](https://docs.micropython.org/en/latest/reference/packages.html)
+documentation.
 
-Other libraries are packages, in which case you'll need to copy the directory instead. For example, to add `collections.defaultdict`, copy `collections/collections/__init__.py` and `collections.defaultdict/collections/defaultdict.py` to a directory named `lib/collections` on your device.
+### On a network-enabled device
 
-Future plans (and new contributor ideas)
-----------------------------------------
+As of MicroPython v1.20 (and nightly builds since October 2022), boards
+with WiFi and Ethernet support include the `mip` package manager.
+
+```py
+>>> import mip
+>>> mip.install("package-name")
+```
+
+### Using `mpremote` from your PC
+
+`mpremote` is the officially-supported tool for interacting with a MicroPython
+device and, since v0.4.0, support for installing micropython-lib packages is
+provided by using the `mip` command.
+
+```bash
+$ mpremote connect /dev/ttyUSB0 mip install package-name
+```
+
+See the [mpremote documentation](https://docs.micropython.org/en/latest/reference/mpremote.html).
+
+### Freeze into your firmware
+
+If you are building your own firmware, all packages in this repository include
+a `manifest.py` that can be included into your board manifest via the
+`require()` command. See [Manifest files](https://docs.micropython.org/en/latest/reference/manifest.html#require) for
+more information.
+
+### Copy the files manually
+
+Many micropython-lib packages are just single-file modules, and you can
+quickly get started by copying the relevant Python file to your device. For
+example, to add the `base64` library, you can directly copy
+`python-stdlib/base64/base64.py` to the `lib` directory on your device.
+
+This can be done using `mpremote`, for example:
+
+```bash
+$ mpremote connect /dev/ttyUSB0 cp python-stdlib/base64/base64.py :/lib
+```
+
+For packages that are implemented as a package directory, you'll need to copy
+the directory instead. For example, to add `collections.defaultdict`, copy
+`collections/collections/__init__.py` and
+`collections-defaultdict/collections/defaultdict.py` to a directory named
+`lib/collections` on your device.
+
+Note that unlike the other three approaches based on `mip` or `manifest.py`,
+you will need to manually resolve dependencies. You can inspect the relevant
+`manifest.py` file to view the list of dependencies for a given package.
+
+## Installing packages from forks
+
+It is possible to use the `mpremote mip install` or `mip.install()` methods to
+install packages built from a
+[fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/about-forks)
+of micropython-lib, if the fork's owner has opted in.
+
+This can be useful to install packages from a pending Pull Request, for example.
+
+First, the owner of the fork must opt-in as described under
+[Publishing packages from forks](CONTRIBUTING.md#publishing-packages-from-forks).
+
+After this has happened, each time someone pushes to a branch in that fork then
+GitHub Actions will automatically publish the packages to a GitHub Pages site.
+
+To install these packages, use commands such as:
+
+```bash
+$ mpremote connect /dev/ttyUSB0 mip install --index https://USERNAME.github.io/micropython-lib/mip/BRANCH_NAME PACKAGE_NAME
+```
+
+Or from a networked device:
+
+```py
+import mip
+mip.install(PACKAGE_NAME, index="https://USERNAME.github.io/micropython-lib/mip/BRANCH_NAME")
+```
+
+(Where `USERNAME`, `BRANCH_NAME` and `PACKAGE_NAME` are replaced with the owner
+of the fork, the branch the packages were built from, and the package name.)
+
+## Contributing
+
+We use [GitHub Discussions](https://github.com/micropython/micropython/discussions)
+as our forum, and [Discord](https://micropython.org/discord) for chat. These
+are great places to ask questions and advice from the community or to discuss your
+MicroPython-based projects.
+
+The [MicroPython Wiki](https://github.com/micropython/micropython/wiki) is
+also used for micropython-lib.
+
+For bugs and feature requests, please [raise an issue](https://github.com/micropython/micropython-lib/issues/new).
+
+We welcome pull requests to add new packages, fix bugs, or add features.
+Please be sure to follow the
+[Contributor's Guidelines & Code Conventions](CONTRIBUTING.md). Note that
+MicroPython is licensed under the [MIT license](LICENSE) and all contributions
+should follow this license.
+
+### Future plans (and new contributor ideas)
+
+* Develop a set of example programs using these packages.
+* Develop more MicroPython packages for common tasks.
+* Expand unit testing coverage.
+* Add support for referencing remote/third-party repositories.
+
+## Notes on terminology
+
+The terms *library*, *package*, and *module* are overloaded and lead to some
+confusion. The interpretation used in by the MicroPython project is that:
+
+A *library* is a collection of installable packages, e.g. [The Python Standard
+  Library](https://docs.python.org/3/library/), or micropython-lib.
+
+A *package* can refer to two things. The first meaning, "library package", is
+something that can be installed from a library, e.g. via `mip` (or `pip` in
+CPython/PyPI). Packages provide *modules* that can be imported. The ambiguity
+here is that the module provided by the package does not necessarily have to
+have the same name, e.g. the `pyjwt` package provides the `jwt` module. In
+CPython, the `pyserial` package providing the `serial` module is another
+common example.
+
+A *module* is something that can be imported. For example, "the *os* module".
+
+A module can be implemented either as a single file, typically also called
+a *module* or "single-file module", or as a *package* (the second meaning),
+which in this context means a directory containing multiple `.py` files
+(usually at least an `__init__.py`).
+
+In micropython-lib, we also have the concept of an *extension package* which
+is a library package that extends the functionality of another package, by
+adding additional files to the same package directory. These packages have
+hyphenated names. For example, the `collections-defaultdict` package extends
+the `collections` package to add the `defaultdict` class to the `collections`
+module.
 
-* Develop a set of example programs using these libraries.
-* Develop more MicroPython libraries for common tasks.

micropython/README.md

@@ -1,13 +1,9 @@
-MicroPython-specific libraries
-==============================
+## MicroPython-specific packages
 
-These are libraries that have been written specifically for use on MicroPython.
+These are packages that have been written specifically for use on MicroPython.
 
-In some cases, the libraries are inspired by or based on equivalent CPython standard libraries, but compatibility varies. The libraries are often named with a "u" prefix.
+Packages in this directory should not have the same name as modules from the Python Standard Library.
 
-Other libraries have been written specifically for MicroPython use cases.
+### Future plans
 
-Future plans
-------------
-
-* More organised directory structure based on library purpose (e.g. drivers, network, etc).
+* More organised directory structure based on purpose (e.g. drivers, network, etc).

micropython/aiorepl/aiorepl.py

@@ -104,13 +104,18 @@ async def task(g=None, prompt="--> "):
             cmd = ""
             while True:
                 b = await s.read(1)
-                c = ord(b)
                 pc = c  # save previous character
+                c = ord(b)
                 pt = t  # save previous time
                 t = time.ticks_ms()
                 if c < 0x20 or c > 0x7E:
                     if c == 0x0A:
-                        # CR
+                        # LF
+                        # If the previous character was also LF, and was less
+                        # than 20 ms ago, this was likely due to CRLF->LFLF
+                        # conversion, so ignore this linefeed.
+                        if pc == 0x0A and time.ticks_diff(t, pt) < 20:
+                            continue
                         sys.stdout.write("\n")
                         if cmd:
                             # Push current command.