Merge pull request #88 from psychomario/development

Various Patches which Complete v1.6; this fixes #85, fixes #83, fixes #82, fixes #80, fixes #79, and fixes #27.

DOCUMENTATION.md

@@ -42,12 +42,20 @@ We have implemented GET and HEAD, as there is no requirement for any other metho
 
 The HEAD method is used by some PXE ROMs to find the Content-Length before the GET is sent.
 
+## NBD
+NBD is similar to NFS in that it can act as a root device for Linux systems. Defined in the [specification](https://github.com/yoe/nbd/blob/master/doc/proto.txt), NBD allows access to block devices over the network by performing read and write requests on the block device itself.
+
+This is different to NFS as it does not act as a filesystem, merely a single file. NBD supports read/write access along with copy-on-write support, both in memory and on disk. Read/write without copy-on-write is potentially dangerous if the file-system layer does not support multiple systems accessing it at the same time. Copy-on-write alleviates these potential problems by providing a volatile layer in which per-client changes are saved. Both the on-disk and in-memory configurations delete changes after the client disconnects, but the in-memory configuration may offer a speed increase as the changes are stored in the system RAM.
+
+WARNING: The use of this option can potentially consume a large amount of RAM; up to the size of the disk image multiplied by the number of connected clients may be used. The same can be said for the on-disk configuration, where this configuration uses disk space rather than memory space. A further configuration option to store the original disk image in memory is provided to potentially allow for read/write speed up.
+
 # PyPXE Services
-The PyPXE library provies the following services for the purpose of creating a Python-based PXE environment: TFTP, HTTP, and DHCP. Each service must be imorted independently as such:
+The PyPXE library provides the following services for the purpose of creating a Python-based PXE environment: TFTP, HTTP, DHCP, and NBD. Each service must be imported independently as such:
 
 * `from pypxe import tftp` or `import pypxe.tftp` imports the TFTP service
 * `from pypxe import dhcp` or `import pypxe.dhcp` imports the DHCP service
 * `from pypxe import http` or `import pypxe.http` imports the HTTP service
+* `from pypxe import nbd` or `import pypxe.nbd` imports the NBD service
 
 **See [`pypxe-server.py`](pypxe-server.py) in the root of the repo for example usage on how to call, define, and setup the services.** When running any Python script that uses these classes, it should be run as a user with root privileges as they bind to interfaces and without root privileges the services will most likely fail to bind properly.
 
@@ -70,7 +78,7 @@ The TFTP server class, __`TFTPD()`__, is constructed with the following __keywor
 |__`ip`__|This is the IP address that the TFTP server will bind to.|`'0.0.0.0'` (so that it binds to all available interfaces)| _string_|
 |__`port`__|This it the port that the TFTP server will run on.|`69` (default port for TFTP)|_int_|
 |__`netboot_directory`__|This is the directory that the TFTP server will serve files from similarly to that of `tftpboot`.|`'.'` (current directory)|_string_|
-|__`mode_debug`__|This indicates whether or not the TFTP server should be started in debug mode or not.|`False`|`bool`|
+|__`mode_debug`__|This indicates whether or not the TFTP server should be started in debug mode or not.|`False`|_bool_|
 |__`logger`__|A [Logger](https://docs.python.org/2/library/logging.html#logger-objects) object used for logging messages, if `None` a local [StreamHandler](https://docs.python.org/2/library/logging.handlers.html#streamhandler) instance will be created.|`None`|[_Logger_](https://docs.python.org/2/library/logging.html#logger-objects)|
 |__`default_retries`__|The number of data retransmissions before dropping a connection.|`3`|_int_|
 |__`timeout`__|The time in seconds before re-sending an un-acknowledged data block.|`5`|_int_|
@@ -86,7 +94,7 @@ from pypxe import dhcp
 import pypxe.dhcp
 ```
 
-###Usage
+### Usage
 The DHCP server class, __`DHCPD()`__, is constructed with the following __keyword arguments__:
 
 |Keyword Argument|Description|Default|Type|
@@ -104,6 +112,8 @@ The DHCP server class, __`DHCPD()`__, is constructed with the following __keywor
 |__`use_ipxe`__|This indicates whether or not iPXE is being used and adjusts itself accordingly.|`False`|_bool_|
 |__`use_http`__|This indicates whether or not the built-in HTTP server is being used and adjusts itself accordingly.|`False`|_bool_|
 |__`mode_proxy`__|This indicates whether or not the DHCP server should be started in ProxyDHCP mode or not.|`False`|_bool_|
+|__`static_config`__|This specifies a static configuration dictionary so that it can give specific leases to specific MAC addresses.|`{}`|_dict_|
+|__`whitelist`__|This indicates whether or not the DHCP server should use the static configuration dictionary as a whitelist; effectively, the DHCP server will only give out leases to those specified in the `static_config` dictionary.|`False`|_bool_|
 |__`mode_debug`__|This indicates whether or not the DHCP server should be started in debug mode or not.|`False`|_bool_|
 |__`logger`__|A [Logger](https://docs.python.org/2/library/logging.html#logger-objects) object used for logging messages, if `None` a local [StreamHandler](https://docs.python.org/2/library/logging.handlers.html#streamhandler) instance will be created.|`None`|[_Logger_](https://docs.python.org/2/library/logging.html#logger-objects)|
 
@@ -126,9 +136,36 @@ The HTTP server class, __`HTTPD()`__, is constructed with the following __keywor
 |__`ip`__|This is the IP address that the HTTP server will bind to.|`'0.0.0.0'` (so that it binds to all available interfaces)|_string_|
 |__`port`__|This it the port that the HTTP server will run on.|`80` (default port for HTTP)|_int_|
 |__`netboot_directory`__|This is the directory that the HTTP server will serve files from similarly to that of `tftpboot`.|`'.'` (current directory)|_string_|
-|__`mode_debug`__|This indicates whether or not the HTTP server should be started in debug mode or not.|`False`|bool|
+|__`mode_debug`__|This indicates whether or not the HTTP server should be started in debug mode or not.|`False`|_bool_|
 |__`logger`__|A [Logger](https://docs.python.org/2/library/logging.html#logger-objects) object used for logging messages, if `None` a local [StreamHandler](https://docs.python.org/2/library/logging.handlers.html#streamhandler) instance will be created.|`None`|[_Logger_](https://docs.python.org/2/library/logging.html#logger-objects)|
 
+## NBD Server `pypxe.nbd`
+
+### Importing
+The NBD service can be imported _one_ of the following two ways:
+```python
+from pypxe import http
+```
+```python
+import pypxe.nbd
+```
+
+### Usage
+The NBD server class, __`NBD()`__, is constructed with the following __keyword arguments__:
+
+|Keyword Argument|Description|Default|Type|
+|---|---|---|---|
+|__`ip`__|This is the IP address that the NBD server will bind to.|`'0.0.0.0'` (so that it binds to all available interfaces)|_string_|
+|__`port`__|This it the port that the NBD server will run on.|`10809` (default port for NBD)|_int_|
+|__`block_device`__|The filename of the block device to be used as the root device.|`''`|_string_|
+|__`write`__|Enable write support on the block device.|`False`|_bool_|
+|__`cow`__|Enable copy-on-write support on the block device.|`True`|_bool_|
+|__`in_mem`__|Enable _in-memory_ copy-on-write support on the block device. `False` causes changes to be stored on disk.|`False`|_bool_|
+|__`copy_to_ram`__|Copy the disk image to RAM when the service starts.|`False`|_bool_|
+|__`mode_debug`__|This indicates whether or not the NBD server should be started in debug mode or not.|`False`|_bool_|
+|__`logger`__|A [Logger](https://docs.python.org/2/library/logging.html#logger-objects) object used for logging messages, if `None` a local [StreamHandler](https://docs.python.org/2/library/logging.handlers.html#streamhandler) instance will be created.|`None`|[_Logger_](https://docs.python.org/2/library/logging.html#logger-objects)|
+
+
 ## Additional Information
 * The function `chr(0)` is used in multiple places throughout the servers. This denotes a `NULL` byte, or `\x00`
 * Python 2.6 does not include the `argparse` module, it is included in the standard library as of 2.7 and newer. The `argparse` module is required to take in command line arguments and `pypxe-server.py` will not run without it.

README.md

@@ -43,12 +43,17 @@ The following are arguments that can be passed to `pypxe-server.py` when running
 |Argument|Description|Default|
 |---|---|---|
 |__`--ipxe`__|Enable iPXE ROM|`False`|
+|__`--no-ipxe`__|Disable iPXE ROM|`True`|
 |__`--http`__|Enable built-in HTTP server|`False`|
+|__`--no-http`__|Disable built-in HTTP server|`True`|
 |__`--dhcp`__|Enable built-in DHCP server|`False`|
 |__`--dhcp-proxy`__|Enable built-in DHCP server in proxy mode (implies `--dhcp`)|`False`|
+|__`--no-dhcp`__|Disable built-in DHCP server|`True`|
+|__`--tftp`__|Enable built-in TFTP server which is enabled by default|`True`|
 |__`--no-tftp`__|Disable built-in TFTP server which is enabled by default|`False`|
-|__`--debug`__|Enable selected services in DEBUG mode; services are selected by passing the name in a comma separated list. **Options are: http, tftp and dhcp** _This adds a level of verbosity so that you can see what's happening in the background._|`''`|
+|__`--debug`__|Enable selected services in DEBUG mode; services are selected by passing the name in a comma separated list. **Options are: http, tftp and dhcp**; one can also prefix an option with `-` to prevent debugging of that service; for example, the following will enable debugging for all services _except_ the DHCP service `--debug all,-dhcp`. _This mode adds a level of verbosity so that you can see what's happening in the background._|`''`|
 |__`--config`__|Load configuration from JSON file. (see [`example_cfg.json`](example_cfg.json))|`None`|
+|__`--static-config`__|Load DHCP lease configuration from JSON file. (see [`example-leases.json`](example-leases.json))|`None`|
 |__`--syslog`__|Specify a syslog server|`None`|
 |__`--syslog-port`__|Specify a syslog server port|`514`|
 
@@ -57,25 +62,39 @@ The following are arguments that can be passed to `pypxe-server.py` when running
 
 |Argument|Description|Default|
 |---|---|---|
-|__`-s DHCP_SERVER_IP`__ or __`--dhcp-server-ip DHCP_SERVER_IP`__|Specify DHCP server IP address|`192.168.2.2`|
-|__`-p DHCP_SERVER_PORT`__ or __`--dhcp-server-port DHCP_SERVER_PORT`__|Specify DHCP server port|`67`|
-|__`-b DHCP_OFFER_BEGIN`__ or __`--dhcp-begin DHCP_OFFER_BEGIN`__|Specify DHCP lease range start|`192.168.2.100`|
-|__`-e DHCP_OFFER_END`__ or __`--dhcp-end DHCP_OFFER_END`__|Specify DHCP lease range end|`192.168.2.150`|
-|__`-n DHCP_SUBNET`__ or __`--dhcp-subnet DHCP_SUBNET`__|Specify DHCP subnet mask|`255.255.255.0`|
-|__`-r DHCP_ROUTER`__ or __`--dhcp-router DHCP_ROUTER`__|Specify DHCP lease router|`192.168.2.1`|
-|__`-d DHCP_DNS`__ or __`--dhcp-dns DHCP_DNS`__|Specify DHCP lease DNS server|`8.8.8.8`|
-|__`-c DHCP_BROADCAST`__ or __`--dhcp-broadcast DHCP_BROADCAST`__|Specify DHCP broadcast address|`'<broadcast>'`|
-|__`-f DHCP_FILESERVER_IP`__ or __`--dhcp-fileserver-ip DHCP_FILESERVER_IP`__|Specify DHCP file server IP address|`192.168.2.2`|
+|__`--dhcp-server-ip DHCP_SERVER_IP`__|Specify DHCP server IP address|`192.168.2.2`|
+|__`--dhcp-server-port DHCP_SERVER_PORT`__|Specify DHCP server port|`67`|
+|__`--dhcp-begin DHCP_OFFER_BEGIN`__|Specify DHCP lease range start|`192.168.2.100`|
+|__`--dhcp-end DHCP_OFFER_END`__|Specify DHCP lease range end|`192.168.2.150`|
+|__`--dhcp-subnet DHCP_SUBNET`__|Specify DHCP subnet mask|`255.255.255.0`|
+| __`--dhcp-router DHCP_ROUTER`__|Specify DHCP lease router|`192.168.2.1`|
+|__`--dhcp-dns DHCP_DNS`__|Specify DHCP lease DNS server|`8.8.8.8`|
+|__`--dhcp-broadcast DHCP_BROADCAST`__|Specify DHCP broadcast address|`'<broadcast>'`|
+|__`--dhcp-fileserver-ip DHCP_FILESERVER_IP`__|Specify DHCP file server IP address|`192.168.2.2`|
+|__`--dhcp-whitelist`__|Only serve clients specified in the static lease file (`--static-config`)|`False`|
 
 
 ##### File Name/Directory Arguments
 
 |Argument|Description|Default|
 |---|---|---|
-|__`-a NETBOOT_DIR`__ or __`--netboot-dir NETBOOT_DIR`__|Specify the local directory where network boot files will be served|`'netboot'`|
-|__`-i NETBOOT_FILE`__ or __`--netboot-file NETBOOT_FILE`__|Specify the PXE boot file name|_automatically set based on what services are enabled or disabled, see [`DOCUMENTATION.md`](DOCUMENTATION.md) for further explanation_|
+|__`--netboot-dir NETBOOT_DIR`__|Specify the local directory where network boot files will be served|`'netboot'`|
+|__`--netboot-file NETBOOT_FILE`__|Specify the PXE boot file name|_automatically set based on what services are enabled or disabled, see [`DOCUMENTATION.md`](DOCUMENTATION.md) for further explanation_|
+
+
+##### Network Block Device Arguments
+|Argument|Description|Default|
+|---|---|---|
+|__`--nbd NBD_BLOCK_DEVICE`__|Specify the block device to be served by NBD and enable NBD. This can be a disk image.|`''`|
+|__`--nbd-write`__|Open the block device for write access. UNSAFE: Multiple clients can cause corruption|`False`|
+|__`--nbd-cow`__|When write is enabled, create a *volatile* file per client with their changes. Clients can write but changes are not shared or kept.|`True (Only applies if write is on)`|
+|__`--nbd-cow-in-mem`__|Client volatile changes are stored in RAM rather than on disk. WARNING: High RAM usage (up to sizeof(block device)*clients)|`False`|
+|__`--nbd-copy-to-ram`__|Disk image is copied to RAM on start to speed up access. Changes are lost when write is used without cow.|`False`|
+|__`--nbd-server`__|The NBD server IP address to bind to|`0.0.0.0`|
+|__`--nbd-port`__|The NBD server port to bind to|`10809`|
+
 
 ## Notes
 * `Core.iso` located in `netboot` is from the [TinyCore Project](http://distro.ibiblio.org/tinycorelinux/) and is provided as an example to network boot from using PyPXE
-* `chainload.kpxe` located in `netboot` is the `undionly.kpxe` from the [iPXE Project](http://ipxe.org/)  
+* `chainload.kpxe` located in `netboot` is the `undionly.kpxe` from the [iPXE Project](http://ipxe.org/)
 * `ldlinux.c32`, `libutil.c32`, `pxelinux.0`, `menu.c32`, and `memdisk` located in `netboot` are from the [SYSLINUX Project](http://www.syslinux.org/) version [6.02](http://www.syslinux.org/wiki/index.php/Syslinux_6_Changelog#Changes_in_6.02)

example_cfg.json

@@ -16,4 +16,11 @@
     "USE_TFTP" : true,
     "USE_DHCP" : true,
     "DHCP_MODE_PROXY" : false,
+    "NBD_BLOCK_DEVICE" : "",
+    "NBD_WRITE" : false,
+    "NBD_COW" : true,
+    "NBD_COW_IN_MEM" : false,
+    "NBD_COPY_TO_RAM" : false,
+    "NBD_SERVER_IP" : "0.0.0.0",
+    "NBD_SERVER_PORT" : 10809,
     "MODE_DEBUG" : ""   }

example_leases.json

@@ -0,0 +1,15 @@
+{
+    "dhcp": {
+        "binding": {
+            "5C:9A:D8:5E:F2:E5": {
+                "dns": [
+                    "8.8.8.8",
+                    "8.8.4.4"
+                ],
+                "ipaddr": "192.168.0.123",
+                "router": "192.168.0.1",
+                "subnet": "255.255.255.0"
+            }
+        }
+    }
+}

netboot/boot.http.nbd.ipxe

@@ -0,0 +1,4 @@
+#!ipxe
+initrd http://192.168.0.11/initramfs-linux.img
+chain http://192.168.0.11/vmlinuz-linux quiet ip=::::::dhcp nbd_host=192.168.0.11 nbd_name=arch.img root=/dev/nbd0
+sanboot